Sparse attention cuts long-context LLM inference cost, but ...
Quick Answer
SparDA, developed by NVIDIA and MIT, enhances long-context LLM inference by introducing a Forecast layer, reducing KV cache growth and improving block selection efficiency.
Quick Take
This results in up to 1.25x prefill and 1.7x decode speedup on models like MiniCPM4.1-8B and NOSA-8B, while maintaining accuracy across multiple benchmarks.
Key Points
- SparDA introduces a fourth projection layer to predict needed KV blocks.
- It adds less than 0.5% parameters, training only the Forecast weights.
- Achieves up to 5.3x higher decode throughput with larger batch sizes.
- Matches or improves accuracy on HELMET, LongBench, and RULER benchmarks.
- Reduces inference costs associated with long-context .
📖 Reader Mode
~1 min readSparse attention cuts long-context LLM inference cost, but two bottlenecks remain: the KV cache still grows with sequence length (offloading it to CPU hits the PCIe wall), and picking which blocks to attend to becomes its own dominant cost at long contexts. SparDA (NVIDIA + MIT) fixes both with one architectural change: a fourth per-layer projection, the Forecast, alongside Q/K/V. It predicts the NEXT layer's needed KV blocks one layer ahead, so block selection and CPU-to-GPU prefetch overlap with the current layer's compute. It adds <0.5% parameters (33.5M on 8B) and trains only the Forecast weights. Headline: on MiniCPM4.1-8B and NOSA-8B, SparDA matches or improves accuracy on HELMET, LongBench, RULER, and a long-reasoning suite - with up to 1.25x prefill and 1.7x decode speedup over the sparse-attention offload baseline, and up to 5.3x higher decode throughput from the larger batches that prefetch makes feasible. Made a short visual breakdown - one diagram per trick. Swipe through. 👇 --- paper - arxiv.org/abs/2606.04511 code - github.com/NVlabs/SparDA full summary pdf - gist.github.com/vukrosic/e2c70… 🔬 Every Sunday I run a hands-on live AI research with 1 on 1 help: skool.com/become-ai-rese…
— Originally published at x.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from WebSearch (Tavily)
See more →全球AI芯片峰会，9月上海见！
The 2026 Global AI Chip Summit will take place in Shanghai on September 22-23, focusing on the evolving AI chip landscape, including the shift from training to inference, the rise of diverse chip technologies, and the restructuring of industry competition. Notable speakers include experts from leading universities and companies, discussing advancements in AI chip architecture and applications.