SENTINEL: Failure-Driven Reinforcement Learning for Training Tool-Using Language Model Agents
Quick Answer
SENTINEL is a failure-driven reinforcement learning framework that enhances tool-using language model agents by turning rollout failures into targeted training tasks.
Quick Take
SENTINEL is a failure-driven reinforcement learning framework that enhances tool-using language model agents by turning rollout failures into targeted training tasks. Tested on Tau2-Bench Retail with Qwen3-4B-Thinking-2507, it improved Pass^1 scores from 66.4 to 74.9, outperforming traditional RL methods on synthetic tasks.
Key Points
- SENTINEL uses a Controller-Proposer-Solver loop to analyze and address model failures.
- The framework generates tasks that specifically target recurring error patterns in agent performance.
- SENTINEL achieved a Pass^1 score improvement of 8.5 points on Tau2-Bench Retail.
- It outperformed traditional reinforcement learning on general synthetic tasks across Pass^k metrics.
- The approach demonstrates a scalable method for enhancing tool-using language model agents.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 12908v1 Announce Type: new Abstract: Language model agents are increasingly effective in solving realistic tasks through multi-turn . However, training reliable tool-using agents remains challenging in practice. While reinforcement learning provides an on-policy paradigm for improving agents from their own environment interactions, its effectiveness depends heavily on the training task distribution.
When tasks are fixed before training, the task distribution can become increasingly mismatched with the policy's evolving capabilities, causing many rollouts to be spent on uninformative tasks. We propose SENTINEL, a failure-driven reinforcement learning framework that turns the Solver's rollout failures into targeted training tasks.
SENTINEL follows a Controller--Proposer--Solver loop: the Controller analyzes failed trajectories and summarizes recurring error patterns, the Proposer generates executable tasks that stress these weaknesses, and the Solver is trained on the targeted tasks. On Tau2-Bench Retail with Qwen3-4B-Thinking-2507, SENTINEL improves Pass\^{}1 from 66. 4 to 74. 9 and outperforms RL on general synthetic tasks across Pass\^{}k metrics.
These results demonstrate that model failures provide an effective and scalable source of targeted training signal for improving tool-using language model agents.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.