AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens
Quick Answer
This paper shows that AbridgeKV compresses generated text into KV cache using learned pause tokens, reducing cache size by 3.5x and improving problem-solving speed by 3.2x on H100 GPUs.
Quick Take
It maintains accuracy on competitive benchmarks with a quarter of the cache compared to full-KV models, outperforming traditional eviction methods.
Key Points
- AbridgeKV uses learned pause tokens to compress generated text into KV cache.
- Achieves 3.5x smaller cache size per sequence compared to full-KV models.
- Improves problem-solving speed by 3.2x on saturated H100 GPUs.
- Maintains accuracy on 446 competition problems with reduced cache.
- Outperforms traditional eviction methods and closest trained compressors.
DeepSignal Analysis
What happened
AbridgeKV introduces a method to compress generated text into KV cache using learned pause tokens. This approach reduces the cache size by 3.5 times and enhances problem-solving speed by 3.2 times on H100 GPUs while maintaining accuracy on competitive benchmarks.
Key evidence
- AbridgeKV compresses generated text into KV entries using learned pause tokens, which do not produce output.
- The method allows for a cache size that is 3.5 times smaller per sequence compared to full-KV models.
- On 446 competition problems, a quarter of the generated cache matches the accuracy of a full-KV model.
Why it matters
The development of AbridgeKV is significant as it addresses the memory limitations associated with long chain-of-thought decoding in AI models. By effectively compressing output, it allows for faster processing and reduces resource consumption, which is crucial for scaling AI applications. This could lead to more efficient use of computational resources in real-world applications.
What to watch
Future developments should focus on the practical implementation of AbridgeKV in various AI systems and its performance across different hardware setups. Additionally, monitoring how this method compares to other emerging techniques in terms of efficiency and accuracy will be important.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time compute being spent. Evicting cache entries does not fix this, because a model never trained on a cache with entries removed generates longer, often up to the length limit. We instead train the model to compress its own output. AbridgeKV compresses each completed block of generated text into the KV entries of a few learned pause tokens. These are extra input tokens with trained embeddings that produce no output. Once the pause tokens have encoded the block, AbridgeKV frees the block's original cache; subsequent tokens attend to the pause tokens' KV entries in its place. Three novel choices make this work: the pause tokens' KV entries take positions inside the block they replace, the cache budget is allocated across layers according to the measured value of each layer's entries, and compression operates on relatively large blocks. On 446 competition problems from AIME, HMMT, AMC, BRUMO, CMIMC and SMT with 32K-token generations, keeping a quarter of the generated cache matches an identically trained full-KV model in accuracy and output length; the strongest published evictor at a matched budget writes far longer, and the closest trained compressor, retrained on the same data, loses 10 points. Served in vLLM, the cache is 3.5x smaller per sequence and a saturated H100 finishes a problem 3.2x faster than with full KV.
| Subjects: | Computation and Language (cs.CL); Machine Learning (cs.LG) |
| Cite as: | arXiv:2610.20641 [cs.CL] |
| (or arXiv:2610.20641v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20641 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziyang Cai [view email]
[v1]
Wed, 7 Oct 2026 17:39:41 UTC (237 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →The "10th Juror": Open-Set Standpoint Screening for Bureaucratic Bias Detection
MARS-Gov introduces a framework for detecting bureaucratic bias in Dutch government documents, achieving a new state-of-the-art F1 score of 0.880. This model outperforms existing zero-shot detectors by 20.2 points and reduces unnecessary interventions to just 2.5%. The framework's dynamic '10th juror' adapts to emerging biases, enhancing legal language processing.