CAST: Game Solvers as Turn-Level Teachers for LLM Agents
Quick Answer
The CAST framework enhances reinforcement learning for LLM agents by using turn-level signals derived from game solvers' state value changes, outperforming baselines in Sokoban, Minesweeper, and Rush Hour.
Quick Take
This approach achieves superior zero-shot performance on ALFWorld and WebShop, demonstrating a significant advancement in generalist decision-making.
Key Points
- CAST converts game solver state value changes into turn-level credit for training.
- Outperformed all baselines in Sokoban, Minesweeper, and Rush Hour evaluations.
- Achieved highest average zero-shot performance on ALFWorld and WebShop.
- Relies on scalar values instead of teacher logits for on-policy distillation.
- Demonstrates a significant advancement in generalist decision-making for LLMs.
DeepSignal Analysis
What happened
The CAST framework was introduced to improve reinforcement learning for large language model (LLM) agents by utilizing turn-level signals from game solvers. This method demonstrated superior performance in games like Sokoban, Minesweeper, and Rush Hour, as well as achieving high zero-shot performance on ALFWorld and WebShop.
Key evidence
- CAST converts state value changes from game solvers into turn-level signals for reinforcement learning, addressing the issue of sparse rewards.
- In tests, CAST outperformed all trained baselines across Sokoban, Minesweeper, and Rush Hour, showing effectiveness in both in-domain and unseen-difficulty evaluations.
- The framework achieved the highest average zero-shot performance on ALFWorld and WebShop, indicating its potential for generalist decision-making.
Why it matters
The introduction of CAST represents a significant advancement in the training of LLMs for complex decision-making tasks. By providing denser feedback through turn-level signals, it addresses a critical limitation of traditional reinforcement learning methods that rely on sparse rewards. This could enhance the applicability of LLMs in various domains requiring long-term planning and decision-making.
Paper Resources
Source Excerpt
Training (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.