TurnOPD: Making On-Policy Distillation Turn-Aware for Efficient Long-Horizon Agent Training
Quick Answer
TurnOPD introduces a novel turn-level budgeting strategy for on-policy distillation, enhancing long-horizon agent training.
Quick Take
By optimizing rollout depth and KL weighting, it achieves superior validation accuracy on benchmarks like ALFWorld and WebShop, outperforming vanilla OPD under equal training budgets.
Key Points
- TurnOPD addresses inefficiencies in vanilla on-policy distillation for long-horizon tasks.
- Utilizes adaptive rollout-depth budgeting to optimize training resources effectively.
- Implements progressive turn-normalized loss budgeting for improved KL supervision.
- Achieves superior validation accuracy on ALFWorld, WebShop, and Multi-Hop Search.
- Advances the accuracy-time frontier beyond traditional OPD methods.
Paper Resources
📖 Reader Mode
~2 min readAbstract:On-policy distillation (OPD) trains a student policy by matching a stronger teacher on the student's own trajectories, offering a promising framework for language agent training. However, its application to long-horizon agentic tasks remains insufficiently explored. We identify two key inefficiencies in vanilla agent OPD: (1) full-horizon rollouts often waste wall-clock resources on tail turns that provide weak and noisy KL supervision, and (2) trajectory-level KL objectives concentrate most of the loss on shallow tokens, leaving deeper decision turns under-trained once initial behaviors are aligned. To address these challenges, we propose TurnOPD, a turn-level budgeting strategy for efficient on-policy distillation of long-horizon agents. TurnOPD consists of two budget controllers: adaptive rollout-depth budgeting, which uses probe-based turn statistics to determine rollout length, and progressive turn-normalized loss budgeting, which gradually shifts KL weighting from token-level to turn-balanced supervision. Experiments on ALFWorld, WebShop, and Multi-Hop Search with task-specialized teacher models show that TurnOPD achieves superior validation accuracy under equal wall-clock training budgets and advances the accuracy--time frontier beyond vanilla OPD.
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.05804 [cs.AI] |
| (or arXiv:2607.05804v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05804 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yuhang Zhou [view email]
[v1]
Tue, 7 Jul 2026 03:56:35 UTC (2,259 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.