VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs
Quick Answer
VarRate introduces a training-free KV codec for long-context LLMs, achieving only 3.5-5.5 accuracy points degradation compared to traditional methods.
Quick Take
It maintains a variable low-rank budget for tokens, outperforming uniform low-rank coding and matching the performance of existing adaptive codecs without training. On LongBench, VarRate stays within 0.8 points of uncompressed models Llama-3.1-8B and Qwen2.5-7B.
Key Points
- VarRate maintains nonzero rank for all tokens, reducing accuracy loss during query-agnostic reuse.
- It outperforms uniform low-rank coding and matches adaptive codecs without requiring training.
- Achieves a maximum accuracy drop of only 3.5-5.5 points under challenging conditions.
- On LongBench, VarRate remains within 0.8 points of uncompressed models across 16 tasks.
- Significantly better than its uniform-rank ablation and competitive with KVzip.
DeepSignal Analysis
What happened
VarRate is a new training-free key-value codec designed for long-context large language models (LLMs). It allocates a variable low-rank budget to tokens based on their query salience, resulting in minimal accuracy degradation compared to traditional methods. VarRate outperforms uniform low-rank coding and matches the performance of adaptive codecs without requiring training.
Key evidence
- VarRate achieves only 3.5-5.5 accuracy points degradation compared to traditional methods, which is significantly lower than the 11-15 points collapse seen in token-selection methods.
- On LongBench, VarRate maintains performance within 0.8 points of uncompressed models Llama-3.1-8B and Qwen2.5-7B while using a matched 20% budget.
- VarRate demonstrates superior performance over uniform low-rank coding and is accuracy-equivalent to KVzip in three out of four settings, with only one-eighth of the prefill overhead.
Why it matters
The development of VarRate addresses a critical bottleneck in LLM inference related to memory management. By maintaining a variable low-rank budget for tokens, it enhances efficiency without sacrificing accuracy. This could lead to more effective deployment of LLMs in real-world applications, especially where memory constraints are a concern. The ability to achieve competitive performance without training also suggests potential cost savings and faster implementation times for developers.
Paper Resources
Source Excerpt
The key-value (KV) cache is the main memory bottleneck in long-context (LLM) inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance from an observation window and evict low-scoring tokens, but eviction is irreversible -- so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere,
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.