Hierarchical Sparse Attention Done Right: Toward Infinite Context Modeling
Quick Answer
This paper shows that The Hierarchical Landmark Sparse (HiLS) Attention mechanism enhances long-context modeling in LLMs by enabling end-to-end chunk selection and retrieval learning, achieving performance comparable to full attention while extrapolating over 64 times the training context length with 90% accuracy.
Quick Take
The Hierarchical Landmark Sparse (HiLS) Attention mechanism enhances long-context modeling in LLMs by enabling end-to-end chunk selection and retrieval learning, achieving performance comparable to full attention while extrapolating over 64 times the training context length with 90% accuracy. This method allows existing full-attention models to transition to HiLS-Attention with minimal retraining, improving efficiency and effectiveness in long-context tasks.
Key Points
- HiLS-Attention achieves performance on par with full attention for in-domain context lengths.
- Extrapolates over 64 times the training context length with 90% retrieval accuracy.
- Existing full-attention models can transition to HiLS-Attention with lightweight pretraining.
- HiLS-Attention optimizes retrieval scores directly with language modeling loss.
- The method breaks the efficiency-performance trade-off in long-context LLMs.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Xiang Hu, Xinyu Wei, Hao Gu, Minshen Zhang, Tian Liang, Huayang Li, Lei Zhu, Yan Wang, Sirui Han, Yushi Bai, Kewei Tu, Haitao Mi, Leo Liang
Abstract:Scaling modern large language models (LLMs) to long contexts is limited by the quadratic computation cost, and poor length extrapolation of dense attention. Chunk-wise sparse attention offers a promising alternative, but all existing methods fall short of full attention because of their inaccurate chunk selection. We propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling (LM) loss. HiLS factorizes attention hierarchically: each query performs attention independently with each retrieved chunk to extract chunk-specific information, and the resulting outputs are fused according to chunk retrieval scores. By incorporating retrieval scores into the forward attention computation, HiLS optimizes them directly with the LM loss, enabling end-to-end retrieval learning and native sparse training. Experimental results show that HiLS-Attention achieves performance comparable to, and in some cases better than, full attention at in-domain context lengths. Meanwhile, HiLS-Attention extrapolates more than $64\times$ the training context length with 90% retrieval accuracy, far beyond full attention. Moreover, existing full-attention models can be converted to HiLS-Attention with lightweight continued pretraining, preserving in-domain performance while acquiring ultra-long-context extrapolation. Together with its sparse KV access and computation, HiLS-Attention breaks the usual efficiency-performance trade-off, enabling long-context LLMs that are both more efficient and more effective on general long-context tasks than their full-attention counterparts.
| Comments: | preprint |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.02980 [cs.CL] |
| (or arXiv:2607.02980v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.02980 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiang Hu [view email]
[v1]
Fri, 3 Jul 2026 05:39:00 UTC (798 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.