LAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
Quick Answer
LAPO introduces a novel self-generated process-supervision method for multi-turn search reasoning in reinforcement learning, achieving an average exact-match score of 0.326 across seven datasets.
Quick Take
LAPO introduces a novel self-generated process-supervision method for multi-turn search reasoning in reinforcement learning, achieving an average exact-match score of 0.326 across seven datasets. This method outperforms the IGPO baseline by 0.053, utilizing backward leave-one-turn attribution to evaluate the contribution of each search turn without requiring additional reward models or teachers.
Key Points
- LAPO replaces retrieval observations with a [DELETE] placeholder to measure policy changes.
- The method achieves an average exact-match score of 0.326 across multiple datasets.
- LAPO outperforms the IGPO baseline by 0.053 in performance metrics.
- Sign-consistency gating retains only normalized process advantages aligned with raw scores.
- No additional reward model or teacher is required for LAPO's implementation.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LAPO, a self-generated process-supervision method based on backward leave-one-turn attribution. For each search turn, LAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder and measures the resulting change in the current policy's mean log-likelihood of the gold answer. This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context. LAPO further applies sign-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores. The method requires no additional reward model, teacher, verifier, or LLM-as-a-Judge. Across seven knowledge-intensive question-answering datasets with local retrieval, LAPO achieves an average exact-match score of 0.326, outperforming the strongest step-reward baseline, IGPO, by 0.053. Ablations show complementary benefits from backward attribution and sign-consistency gating, demonstrating that policy-derived retrospective attribution can provide effective process supervision for multi-turn search agents.
| Comments: | 20 pages, 5 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.13501 [cs.AI] |
| (or arXiv:2607.13501v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.13501 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Qiang Zhu [view email]
[v1]
Wed, 15 Jul 2026 06:55:28 UTC (313 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Automatic Ordinary Differential Equations Discovery For Biological Systems Using Large Language Model Powered Agentic System
The MEDA system utilizes large language models and symbolic regression to autonomously discover ordinary differential equations for biological systems, achieving strong structural recovery and biologically plausible models. It outperforms existing methods by integrating domain knowledge and mechanistic constraints, demonstrating effective retrieval and extrapolation capabilities.