Recall Before You Rank: Similarity-Guided Top-$K$ Reuse for Efficient Long-Context Attention
Quick Answer
The ReTopK method enhances dynamic Top-K attention by reusing historical query-support pairs, achieving a 3.07x speedup in attention computation while maintaining a mere 0.50% increase in perplexity over Exact Top-K at 128K contexts.
Quick Take
This approach significantly improves efficiency for long-context decoding tasks, outperforming existing approximate methods in benchmarks like PG19 and LongBench.
Key Points
- ReTopK reduces selector cost by reusing historical retrieval decisions.
- Achieves the lowest PG19 perplexity and highest LongBench scores among evaluated methods.
- At 128K contexts with K=512, incurs only a 0.50% perplexity increase.
- Accelerates attention computation by 3.07 times compared to Exact Top-K.
- Maintains a bounded cache of historical query-support pairs for efficiency.
DeepSignal Analysis
What happened
The ReTopK method improves dynamic Top-K attention by reusing historical query-support pairs, achieving a 3.07x speedup in attention computation. It incurs only a 0.50% increase in perplexity over Exact Top-K at 128K contexts, enhancing efficiency in long-context decoding tasks.
Key evidence
- ReTopK maintains a bounded cache of historical query-support pairs and retrieves similar cached queries for new queries.
- At 128K contexts with K=512, ReTopK achieves a 3.07x acceleration in attention computation while only increasing perplexity by 0.50% compared to Exact Top-K.
- ReTopK outperforms existing approximate methods in benchmarks such as PG19 and LongBench, achieving the lowest perplexity and highest scores among evaluated methods.
Why it matters
ReTopK addresses the inefficiencies of sparse attention mechanisms in long-context decoding by leveraging historical data, which could lead to significant performance improvements in natural language processing tasks. The method's ability to maintain low perplexity while enhancing speed is crucial for applications requiring real-time processing of large contexts.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries. However, identifying this subset still requires scoring the current query against the full KV cache and performing global Top-$K$ selection, leaving selector cost linear in context length and limiting the practical efficiency of sparse attention for long-context decoding. In this paper, we introduce ReTopK, a training-free method that accelerates dynamic Top-$K$ attention by reusing historical retrieval decisions. ReTopK builds on the observation that similar queries often attend to overlapping supports and that partially overlapping supports can still preserve most of the Exact Top-$K$ attention mass. For each attention head, it maintains a bounded cache of historical query--support pairs, retrieves the most similar cached queries for each new query, unions their stored supports with a recent window, and reranks only the resulting compact candidate set using exact current-query scores. A similarity-based fallback invokes full-history Exact Top-$K$ when reuse is unreliable, while periodic exact refreshes limit cache drift. ReTopK retains the complete KV cache and reuses only selected indices, rather than historical scores, attention weights, or outputs. Across 16K--128K contexts, ReTopK achieves the lowest PG19 perplexity and the highest NIAH and LongBench scores among the evaluated approximate methods. At 128K with $K=512$, ReTopK incurs only a 0.50\% perplexity increase over Exact Top-$K$ while accelerating attention computation by $3.07\times$.
| Comments: | 9 pages, 9 figures, and 5 tables |
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2607.27692 [cs.CL] |
| (or arXiv:2607.27692v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.27692 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Wenshuai Yao [view email]
[v1]
Thu, 30 Jul 2026 05:25:23 UTC (1,202 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.