ART: Attention Run-time Termination for Efficient Large Language Model Decoding
Quick Take
The paper introduces Attention Run-time Termination (ART), a mechanism that enhances large language model (LLM) decoding efficiency by terminating unnecessary Key-Value (KV) accesses, achieving a 20% increase in generation throughput on LongBench benchmarks without sacrificing accuracy.
Key Points
- ART tracks attention outputs during execution to optimize KV cache access.
- Achieves 20% higher throughput in large batch sizes compared to state-of-the-art methods.
- Maintains comparable accuracy while reducing memory bandwidth constraints.
- Seamlessly integrates with existing key-based KV cache management techniques.
- Proven effective on LongBench benchmarks for large language models.
Article Excerpt
From source RSS / original summaryarXiv:2606. 00024v1 Announce Type: new Abstract: Long-context decoding in Large Language Models (LLMs) is severely constrained by the memory bandwidth required to fetch the extensive Key-Value (KV) cache. Most existing KV management methods rely on key-only pruning before decoding, despite the evidence that attention outputs depend jointly on keys and values, as incorporating values in their methods incurs prohibitive additional overhead.
In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. This design makes ART orthogonal to existing key-based KV cache management methods, enabling seamless integration with them.
Experiments on LongBench benchmarks show that ART achieves 20% higher generation throughput in large batch size than state-of-the-art baseline while maintaining comparable accuracy.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.