CompactAttention: Accelerating Chunked Prefill with Block-Union KV Selection
Quick Answer
CompactAttention introduces a novel chunked-prefill attention mechanism that enhances efficiency for long-context large language models like LLaMA-3.1-8B-Instruct.
Quick Take
CompactAttention introduces a novel chunked-prefill attention mechanism that enhances efficiency for long-context large language models like LLaMA-3.1-8B-Instruct. By employing Block-Union KV Selection, it achieves up to 2.72x speedup while maintaining accuracy on the RULER benchmark, addressing limitations of existing sparse attention methods.
Key Points
- CompactAttention uses Block-Union KV Selection for efficient attention computation.
- Achieves up to 2.72x speedup at 128K context length on LLaMA-3.1-8B-Instruct.
- Maintains accuracy close to dense attention on the RULER benchmark.
- Addresses inefficiencies of existing sparse attention methods in chunked prefill.
- Eliminates explicit KV-copy overhead while preserving query-specific KV entries.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Chunked prefill has become a widely adopted serving strategy for long-context large language models, but efficient attention computation in this regime remains challenging. Existing sparse attention methods are primarily designed for one-shot prefill and do not translate efficiently to chunked prefill: block-sparse kernels lose efficiency when the query length is limited by the chunk size, while fine-grained pattern search becomes costly when repeated over the accumulated KV cache at every chunk. QUOKA, a recent method that directly targets chunked prefill, avoids sparse-kernel overhead but relies on query-subsampled, token-level KV selection, which can miss query-specific KV entries and introduce explicit KV-copy overhead. To address these limitations, we propose CompactAttention, a chunked-prefill attention mechanism based on Block-Union KV Selection. CompactAttention treats 2D block-sparse masks as KV-selection signals rather than direct sparse-kernel execution plans, and converts them into GQA-aware per-group KV block tables through Q-block union and intra-group union. This construction produces the minimal block tables that preserve all KV blocks selected by the input masks under paged execution constraints, enabling selected KV blocks to be accessed in place without explicit KV compaction. On LLaMA-3.1-8B-Instruct, CompactAttention maintains accuracy close to dense attention on the RULER benchmark while delivering up to 2.72$\times$ attention speedup at 128K context length under chunked prefill.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.16839 [cs.CL] |
| (or arXiv:2605.16839v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16839 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiwon Song [view email]
[v1]
Sat, 16 May 2026 06:47:41 UTC (521 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.