LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
Quick Answer
LAPO introduces a novel self-generated process-supervision method for multi-turn search reasoning in reinforcement learning, achieving an average exact-match score of 0.326 across seven datasets.
Quick Take
This method outperforms the IGPO baseline by 0.053, utilizing backward leave-one-turn attribution to evaluate the contribution of each search turn without requiring additional reward models or teachers.
Key Points
- LAPO replaces retrieval observations with a [DELETE] placeholder to measure policy changes.
- The method achieves an average exact-match score of 0.326 across multiple datasets.
- LAPO outperforms the IGPO baseline by 0.053 in performance metrics.
- Sign-consistency gating retains only normalized process advantages aligned with raw scores.
- No additional reward model or teacher is required for LAPO's implementation.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LAPO, a self-generated process-supervision method based on backward leave-one-turn attribution. For each search turn, LAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder and measures the resulting change in the current policy's mean log-likelihood of the gold answer. This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context. LAPO further applies sign-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores. The method requires no additional reward model, teacher, verifier, or LLM-as-a-Judge. Across seven knowledge-intensive question-answering datasets with local retrieval, LAPO achieves an average exact-match score of 0.326, outperforming the strongest step-reward baseline, IGPO, by 0.053. Ablations show complementary benefits from backward attribution and sign-consistency gating, demonstrating that policy-derived retrospective attribution can provide effective process supervision for multi-turn search agents.
| Comments: | 20 pages, 5 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.13501 [cs.AI] |
| (or arXiv:2607.13501v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13501 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiang Zhu [view email]
[v1]
Wed, 15 Jul 2026 06:55:28 UTC (313 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.