Recall Before You Rank: Similarity-Guided Top-$K$ Reuse for Efficient Long-Context Attention
Quick Answer
The ReTopK method enhances dynamic Top-K attention by reusing historical query-support pairs, achieving a 3.07x speedup in attention computation while maintaining a mere 0.50% increase in perplexity over Exact Top-K at 128K contexts.
Quick Take
This approach significantly improves efficiency for long-context decoding tasks, outperforming existing approximate methods in benchmarks like PG19 and LongBench.
Key Points
- ReTopK reduces selector cost by reusing historical retrieval decisions.
- Achieves the lowest PG19 perplexity and highest LongBench scores among evaluated methods.
- At 128K contexts with K=512, incurs only a 0.50% perplexity increase.
- Accelerates attention computation by 3.07 times compared to Exact Top-K.
- Maintains a bounded cache of historical query-support pairs for efficiency.
DeepSignal Analysis
What happened
The ReTopK method improves dynamic Top-K attention by reusing historical query-support pairs, achieving a 3.07x speedup in attention computation. It incurs only a 0.50% increase in perplexity over Exact Top-K at 128K contexts, enhancing efficiency in long-context decoding tasks.
Key evidence
- ReTopK maintains a bounded cache of historical query-support pairs and retrieves similar cached queries for new queries.
- At 128K contexts with K=512, ReTopK achieves a 3.07x acceleration in attention computation while only increasing perplexity by 0.50% compared to Exact Top-K.
- ReTopK outperforms existing approximate methods in benchmarks such as PG19 and LongBench, achieving the lowest perplexity and highest scores among evaluated methods.
Why it matters
ReTopK addresses the inefficiencies of sparse attention mechanisms in long-context decoding by leveraging historical data, which could lead to significant performance improvements in natural language processing tasks. The method's ability to maintain low perplexity while enhancing speed is crucial for applications requiring real-time processing of large contexts.
What to watch
Paper Resources
Source Excerpt
Top-$K$ sparse attention reduces the cost of Softmax and value aggregation by attending to only a small subset of key--value (KV) entries. However, identifying this subset still requires scoring the current query against the full KV cache and performing global Top-$K$ selection, leaving selector cost linear in context length and limiting the practical efficiency of sparse attention for long-context decoding. In this paper, we introduce ReTopK, a training-free method that accelerates dynamic Top-
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.