KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling
Quick Answer
KV-PRM introduces an efficient process reward model that leverages KV cache to reduce scoring costs from O(L^2) to O(L), significantly enhancing multi-agent test-time scaling.
Quick Take
KV-PRM introduces an efficient process reward model that leverages KV cache to reduce scoring costs from O(L^2) to O(L), significantly enhancing test-time scaling. It outperforms traditional text-based PRMs across benchmarks like MATH and GSM8K, achieving up to 5,000x reduction in scoring FLOPs and 37x reduction in latency.
Key Points
- KV-PRM eliminates heavy text re-encoding by directly using KV cache.
- Scoring cost reduction from O(L^2) to O(L) enhances efficiency.
- Achieves up to 5,000x reduction in scoring FLOPs compared to text-PRMs.
- Demonstrates a 37x reduction in latency and 34x in memory footprint.
- Empirically outperforms text-PRMs across various TTS methods.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs' application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM's generation phase. By processing a single "verify token" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O(L^2) to O(L). We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KV-PRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000x reduction in scoring FLOPs, a 37x reduction in latency, and a 34x reduction in per-sequence memory footprint compared to text-based PRMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.09153 [cs.AI] |
| (or arXiv:2607.09153v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09153 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peng Kuang [view email]
[v1]
Fri, 10 Jul 2026 07:16:43 UTC (212 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Adversarial Social Epistemology for Assemblies of Humans and Large Language Models
The paper introduces Adversarial Social Epistemology (ASE) to analyze how agents manipulate trust in public communications, highlighting mechanisms that undermine the reliability of testimony and inference. It critiques existing frameworks like epistemic bubbles and misinformation diffusion, proposing a new language for understanding trust breaches and auditing inferential chains in densely interactive environments involving humans and large language models.