SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
Quick Answer
SENTINEL is a failure-driven reinforcement learning framework that enhances tool-using language model agents by turning rollout failures into targeted training tasks.
Quick Take
Tested on Tau2-Bench Retail with Qwen3-4B-Thinking-2507, it improved Pass^1 scores from 66.4 to 74.9, outperforming traditional RL methods on synthetic tasks.
Key Points
- SENTINEL uses a Controller-Proposer-Solver loop to analyze and address model failures.
- The framework generates tasks that specifically target recurring error patterns in agent performance.
- SENTINEL achieved a Pass^1 score improvement of 8.5 points on Tau2-Bench Retail.
- It outperformed traditional reinforcement learning on general synthetic tasks across Pass^k metrics.
- The approach demonstrates a scalable method for enhancing tool-using language model agents.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Ziyi Wang, Yuxuan Lu, Yimeng Zhang, Qun Liu, Chen Luo, Jiri Gesi, Hanqing Lu, Yisi Sang, Manling Li, Jing Huang, Dakuo Wang
Abstract:Language model agents are increasingly effective in solving realistic tasks through multi-turn tool use. However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution. When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks. SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass\^{}1 from 66.4 to 74.9 and outperforms RL on general synthetic tasks across Pass\^{}k metrics. These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2606.12908 [cs.CL] |
| (or arXiv:2606.12908v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2606.12908 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ziyi Wang [view email]
[v1]
Thu, 11 Jun 2026 05:06:50 UTC (2,130 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.