A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
Quick Answer
SparseSpec-L introduces a training-free self-speculative decoding framework that enhances long-context inference efficiency by utilizing a dynamically sparsified KV cache.
Quick Take
This method allows for significant end-to-end acceleration over traditional autoregressive decoding, maintaining output distribution integrity while optimizing speculation length based on an online entropy controller.
Key Points
- SparseSpec-L reduces memory-bandwidth bottlenecks in inference.
- The framework recycles attention statistics to recall critical historical tokens.
- Experiments show consistent acceleration across multiple long-context tasks.
- The method achieves speedup over autoregressive decoding without sacrificing output quality.
- An online controller dynamically adjusts speculation length for efficiency.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.
| Comments: | 9 pages, 4 figures, subbmited to AAAI 2027 |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.27735 [cs.CL] |
| (or arXiv:2607.27735v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.27735 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yu Guo [view email]
[v1]
Thu, 30 Jul 2026 06:16:56 UTC (1,048 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.