CAST: Game Solvers as Turn-Level Teachers for LLM Agents
Quick Answer
The CAST framework enhances reinforcement learning for LLM agents by using turn-level signals derived from game solvers' state value changes, outperforming baselines in Sokoban, Minesweeper, and Rush Hour.
Quick Take
This approach achieves superior zero-shot performance on ALFWorld and WebShop, demonstrating a significant advancement in generalist decision-making.
Key Points
- CAST converts game solver state value changes into turn-level credit for training.
- Outperformed all baselines in Sokoban, Minesweeper, and Rush Hour evaluations.
- Achieved highest average zero-shot performance on ALFWorld and WebShop.
- Relies on scalar values instead of teacher logits for on-policy distillation.
- Demonstrates a significant advancement in generalist decision-making for LLMs.
DeepSignal Analysis
What happened
The CAST framework was introduced to improve reinforcement learning for large language model (LLM) agents by utilizing turn-level signals from game solvers. This method demonstrated superior performance in games like Sokoban, Minesweeper, and Rush Hour, as well as achieving high zero-shot performance on ALFWorld and WebShop.
Key evidence
- CAST converts state value changes from game solvers into turn-level signals for reinforcement learning, addressing the issue of sparse rewards.
- In tests, CAST outperformed all trained baselines across Sokoban, Minesweeper, and Rush Hour, showing effectiveness in both in-domain and unseen-difficulty evaluations.
- The framework achieved the highest average zero-shot performance on ALFWorld and WebShop, indicating its potential for generalist decision-making.
Why it matters
The introduction of CAST represents a significant advancement in the training of LLMs for complex decision-making tasks. By providing denser feedback through turn-level signals, it addresses a critical limitation of traditional reinforcement learning methods that rely on sparse rewards. This could enhance the applicability of LLMs in various domains requiring long-term planning and decision-making.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Yu Wang, Yi-Kai Zhang, Wentao Shi, Ziang Ye, Yuchun Miao, Yueqing Sun, Qi Gu, Xunliang Cai, Lan-Zhe Guo, Han-Jia Ye, Fuli Feng
Abstract:Training large language models (LLMs) to act in long-horizon games is a promising step toward generalist decision-making, yet reinforcement learning with verifiable rewards (RLVR) relies on sparse final rewards that reveal little about which decisions determine success. Denser process signals could supply this missing turn-level credit, but existing sources are hard to keep both cheap and accurate. We observe that changes in a game solver's state value reveal whether an action advances the state toward success. Building on this insight, we propose CAST (Credit Assignment from Solver Teachers), which converts these value changes into solver advantages and injects them into RLVR as turn-level signals. We further show that, under a soft-optimal solver assumption, maximizing the solver advantage is equivalent to on-policy distillation from the solver, requiring only scalar values rather than teacher logits. Across Sokoban, Minesweeper, and Rush Hour, CAST outperforms all trained baselines on every game under both in-domain and unseen-difficulty evaluation and achieves the highest average zero-shot performance on ALFWorld and WebShop. Our code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.25308 [cs.CL] |
| (or arXiv:2607.25308v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.25308 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yu Wang [view email]
[v1]
Tue, 28 Jul 2026 05:39:01 UTC (598 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.