TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
Quick Answer
TurnOPD introduces a novel turn-level budgeting strategy for on-policy distillation, enhancing long-horizon agent training.
Quick Take
TurnOPD introduces a novel turn-level budgeting strategy for on-policy distillation, enhancing long-horizon agent training. By optimizing rollout depth and KL weighting, it achieves superior validation accuracy on benchmarks like ALFWorld and WebShop, outperforming vanilla OPD under equal training budgets.
Key Points
- TurnOPD addresses inefficiencies in vanilla on-policy distillation for long-horizon tasks.
- Utilizes adaptive rollout-depth budgeting to optimize training resources effectively.
- Implements progressive turn-normalized loss budgeting for improved KL supervision.
- Achieves superior validation accuracy on ALFWorld, WebShop, and Multi-Hop Search.
- Advances the accuracy-time frontier beyond traditional OPD methods.
Paper Resources
📖 Reader Mode
~2 min readAbstract:On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.05804 [cs.AI] |
| (or arXiv:2607.05804v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05804 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhang Zhou [view email]
[v1]
Tue, 7 Jul 2026 03:56:35 UTC (2,259 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.