Dual Dimensionality for Local and Global Attention
Quick Answer
The study introduces Distance-Adaptive Representation (DAR) for decoder-only Transformers, optimizing attention by using richer representations for local tokens and reduced dimensions for distant ones.
Quick Take
This approach maintains performance comparable to full-dimensional baselines across various model sizes (70M to 410M parameters) while enabling significant reductions in KV cache during inference.
Key Points
- DAR maintains full-dimensional representations for local tokens while reducing dimensions for distant ones.
- Performance closely matches full-dimensional baselines across models with 70M to 410M parameters.
- Uniform dimensionality reduction across tokens leads to worse performance outcomes.
- The findings challenge the assumption of uniform key and value dimensionality in attention mechanisms.
- This approach enables further reductions in KV cache during inference.
Paper Resources
Source Excerpt
Decoder-only Transformers compute attention over the KV cache of preceding tokens. Keys (and Values) are typically represented with the same dimensionality, regardless of its distance from the prediction target. In natural language, however, the next word is most strongly influenced by the immediately preceding tokens. We hypothesize that local and distant tokens impose asymmetric demands on representational capacity: local tokens are more critical for predicting immediate outputs and thus requi
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.