RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents
Quick Answer
This paper shows that RODS (Reward-driven Online Data Synthesis) addresses the depletion of informative samples in multi-turn tool-use reinforcement learning by synthesizing new data based on reward variance.
Quick Take
It achieves comparable performance to a 17K-sample offline pipeline using only 800 samples, requiring 20x fewer trajectories and dynamically evolving with the policy.
Key Points
- RODS synthesizes new samples using reward variance as a boundary detector.
- It maintains an active training pool of approximately 800 samples.
- RODS achieves similar performance to a 17K-sample offline pipeline.
- Requires roughly 20x fewer trajectories than traditional methods.
- Dynamic replay buffer evolves alongside the policy during training.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Multi-turn tool-use RL is bottlenecked by the rapid depletion of informative samples in static datasets. We observe that the gradient signal in GRPO concentrates on tasks with the highest rollout reward variance, a consequence of the Popoviciu upper bound. Consequently, samples near the agent's capability boundary -- where successes and failures are roughly balanced -- contribute disproportionately large policy gradients. As training progresses, this boundary continuously shifts, which gradually depletes the pool of informative samples in a static dataset. We propose RODS (Reward-driven Online Data Synthesis) to resolve this depletion. RODS closes the loop between RL training and data generation by repurposing the progress reward variance as a practical, zero-cost boundary detector that requires no extra inference beyond the rollouts already computed for training. It continuously identifies such boundary samples, synthesizes new multi-turn variants matching their structural complexity (e.g., API topology and dependency depth) via a skill-aligned resampling pipeline, and manages a dynamic replay buffer that co-evolves with the policy. Starting from 400 human seeds and maintaining an active training pool of ~800 samples, RODS achieves comparable performance to a 17K-sample offline pipeline while requiring roughly 20x fewer trajectories, and improves over fixed-data RL and environment augmentation in our controlled setting.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.19047 [cs.AI] |
| (or arXiv:2606.19047v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.19047 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ruishan Fang [view email]
[v1]
Wed, 17 Jun 2026 13:13:32 UTC (936 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.