Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps
Quick Answer
This paper shows that RTPurbo transforms full-attention large language models into highly sparse models with minimal adaptation, achieving up to 9.36× prefill speedup and 2.01× decode speedup while maintaining near-lossless accuracy.
Quick Take
RTPurbo transforms full-attention large language models into highly sparse models with minimal adaptation, achieving up to 9.36× prefill speedup and 2.01× decode speedup while maintaining near-lossless accuracy. This method leverages intrinsic sparsity and dynamic token selection, significantly enhancing efficiency in long-context inference without costly pretraining.
Key Points
- RTPurbo retains full KV cache only for retrieval heads, optimizing attention.
- Dynamic top-p selection outperforms fixed top-k in token retrieval efficiency.
- Achieves substantial efficiency gains in long-context benchmarks and reasoning tasks.
- Demonstrates significant speed improvements with minimal training steps.
- No expensive native sparse pretraining required for strong sparse inference.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-$p$ selection more suitable than fixed top-$k$ sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36$\times$ prefill speedup at 1M context and about a 2.01$\times$ decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.
| Comments: | 20 pages, 9 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.16928 [cs.CL] |
| (or arXiv:2605.16928v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16928 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yanke Zhou [view email]
[v1]
Sat, 16 May 2026 10:51:58 UTC (2,772 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.