KV-PRM: Efficient Process Reward Modeling via KV-Cache Transfer for Multi-Agent Test-Time Scaling
Quick Answer
KV-PRM introduces an efficient process reward model that leverages KV cache to reduce scoring costs from O(L^2) to O(L), significantly enhancing multi-agent test-time scaling.
Quick Take
It outperforms traditional text-based PRMs across benchmarks like MATH and GSM8K, achieving up to 5,000x reduction in scoring FLOPs and 37x reduction in latency.
Key Points
- KV-PRM eliminates heavy text re-encoding by directly using KV cache.
- Scoring cost reduction from O(L^2) to O(L) enhances efficiency.
- Achieves up to 5,000x reduction in scoring FLOPs compared to text-PRMs.
- Demonstrates a 37x reduction in latency and 34x in memory footprint.
- Empirically outperforms text-PRMs across various TTS methods.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Process Reward Models (PRMs) have been proven to be highly effective in guiding test-time scaling (TTS) methods, which significantly boost the capabilities of LLM-based multi-agent systems. However, existing PRMs are text-based: they re-encode the entire trajectory text from scratch. In long multi-agent rollouts, the scoring cost, growing quadratically with respect to sequence length L, creates a severe computational bottleneck, severely limiting PRMs' application in long-context scenarios. To resolve this, we introduce KV-PRM, a highly efficient process reward model that eliminates the heavy text re-encoding by directly reading the KV cache produced naturally during the LLM's generation phase. By processing a single "verify token" against the pre-existing KV cache, KV-PRM reduces the scoring cost from O(L^2) to O(L). We formally prove that the KV cache contains strictly greater information capacity than text, and is more efficient for downstream reward modeling. Empirically, across the MATH, GSM8K, and AIME benchmarks, KV-PRM matches or strictly outperforms text-PRMs under various TTS methods such as Beam Search, MCTS, and Weighted Voting, with up to a 5,000x reduction in scoring FLOPs, a 37x reduction in latency, and a 34x reduction in per-sequence memory footprint compared to text-based PRMs.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.09153 [cs.AI] |
| (or arXiv:2607.09153v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.09153 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Peng Kuang [view email]
[v1]
Fri, 10 Jul 2026 07:16:43 UTC (212 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.