Benchmarking KV-Cache Optimizations across Task Quality and System Performance for Long-Context Serving
Quick Answer
This paper benchmarks KV-cache optimizations like KIVI, TurboQuant, SnapKV, and CaM for long-context workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3.
Quick Take
This paper benchmarks KV-cache optimizations like KIVI, TurboQuant, SnapKV, and CaM for long-context workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. Results indicate that KIVI4 offers stable quality, SnapKV excels in throughput, and CaM shows sensitivity in QA tasks, emphasizing the need for workload-aware KV-cache selection.
Key Points
- Benchmark evaluates KV-cache optimizations on multi-document QA and summarization tasks.
- KIVI4 shows the most consistent quality across different models evaluated.
- SnapKV achieves the highest throughput for long-context workloads.
- CaM provides significant improvements in specific QA tasks but is workload-sensitive.
- Compression ratio alone does not predict overall performance effectively.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language model serving is increasingly limited by KV-cache growth under long-context workloads, yet existing KV-cache compression techniques are difficult to compare because they were evaluated on different models, tasks, budgets, and serving stacks. This paper presents a workload-aware benchmark of representative KV-cache optimization mechanisms spanning quantization, pruning, and merging, including KIVI, TurboQuant, SnapKV, and CaM, evaluated on LongBench-style multi-document QA, single-document QA, few-shot learning, and summarization workloads using Llama-3.1-8B-Instruct and Mistral-7B-Instruct-v0.3. The benchmark measures task quality, mean output throughput, mean time-to-first-token, and realized compression ratio across context-length buckets. The results show that the compression ratio alone is a poor predictor of end-to-end performance. KIVI4 provides the most stable quality across models, SnapKV delivers the strongest long-context throughput, and CaM yields large gains on selected QA workloads but exhibits substantial workload sensitivity in both quality and realized compression ratio. These findings motivate workload-aware selection of KV-cache mechanisms rather than one-size-fits-all compression and provide deployment guidance for long-context serving systems.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2607.05399 [cs.CL] |
| (or arXiv:2607.05399v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05399 arXiv-issued DOI via DataCite |
Submission history
From: Nikita Agrawal [view email]
[v1]
Sun, 3 May 2026 11:50:41 UTC (402 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.