Sparse attention cuts long-context LLM inference cost, but ...
Quick Answer
SparDA, developed by NVIDIA and MIT, enhances long-context LLM inference by introducing a Forecast layer, reducing KV cache growth and improving block selection efficiency.
Quick Take
SparDA, developed by NVIDIA and MIT, enhances long-context LLM inference by introducing a Forecast layer, reducing KV cache growth and improving block selection efficiency. This results in up to 1.25x prefill and 1.7x decode speedup on models like MiniCPM4.1-8B and NOSA-8B, while maintaining accuracy across multiple benchmarks.
Key Points
- SparDA introduces a fourth projection layer to predict needed KV blocks.
- It adds less than 0.5% parameters, training only the Forecast weights.
- Achieves up to 5.3x higher decode throughput with larger batch sizes.
- Matches or improves accuracy on HELMET, LongBench, and RULER benchmarks.
- Reduces inference costs associated with long-context LLMs.
📖 Reader Mode
~1 min readSparse attention cuts long-context LLM inference cost, but two bottlenecks remain: the KV cache still grows with sequence length (offloading it to CPU hits the PCIe wall), and picking which blocks to attend to becomes its own dominant cost at long contexts. SparDA (NVIDIA + MIT) fixes both with one architectural change: a fourth per-layer projection, the Forecast, alongside Q/K/V. It predicts the NEXT layer's needed KV blocks one layer ahead, so block selection and CPU-to-GPU prefetch overlap with the current layer's compute. It adds <0.5% parameters (33.5M on 8B) and trains only the Forecast weights. Headline: on MiniCPM4.1-8B and NOSA-8B, SparDA matches or improves accuracy on HELMET, LongBench, RULER, and a long-reasoning suite - with up to 1.25x prefill and 1.7x decode speedup over the sparse-attention offload baseline, and up to 5.3x higher decode throughput from the larger batches that prefetch makes feasible. Made a short visual breakdown - one diagram per trick. Swipe through. 👇 --- paper - arxiv.org/abs/2606.04511 code - github.com/NVlabs/SparDA full summary pdf - gist.github.com/vukrosic/e2c70… 🔬 Every Sunday I run a hands-on live AI research with 1 on 1 help: skool.com/become-ai-rese…
— Originally published at x.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from WebSearch (Tavily)
See more →Stop just chatting with AI. Learn to build production-ready software in ...
The 2026 Bootcamp offers hands-on training in building production-ready software using Generative AI, LLM applications, and AI agents, emphasizing practical skills over casual interaction with AI. Participants will learn to develop applications like Cursor AI, preparing them for real-world challenges in AI development.
