CAST: Non-Privileged Clipped Asymmetric Self-Teaching with Advantage Flipping for GRPO
Quick Answer
CAST introduces a novel answer-free self-distillation method for Group Relative Policy Optimization (GRPO) in reinforcement learning with verifiable rewards (RLVR).
Quick Take
It enhances token-level advantages based on trajectory correctness, improving mathematical reasoning performance while maintaining a lightweight training objective. This approach addresses the limitations of previous self-distillation methods by allowing for bounded advantages in zero-variance groups.
Key Points
- CAST maintains the objective while using a stop-gradient self-teacher for token-level advantages.
- The method allows teacher-negative tokens in correct trajectories to receive negative advantages.
- Experiments demonstrate improved RLVR training for mathematical reasoning tasks.
- CAST does not require reference-solution-conditioned teacher scoring, simplifying the training process.
- Zero-variance groups can now contribute verifier-signed token feedback through bounded advantages.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Reinforcement learning with verifiable rewards (RLVR), especially Group Relative Policy Optimization (GRPO), has been widely used to improve reasoning in large language models. However, outcome-level rewards provide only sparse supervision, and group-relative advantages vanish when all sampled trajectories for a prompt are either correct or incorrect. On-Policy Self-Distillation (OPSD) offers dense token-level guidance, but its token preferences are not necessarily aligned with trajectory correctness; empirical diagnostics show that OPSD signals behave differently on correct and incorrect rollouts, with teacher-positive and teacher-negative gap signals exhibiting different noise profiles. These diagnostics are conducted under an OPSD-style privileged teacher context for analysis only, whereas CAST training uses answer-free self-teacher this http URL by these observations, this work proposes CAST, an answer-free self-distillation method for GRPO-style RLVR. CAST keeps the verifier-grounded GRPO objective, but uses a stop-gradient self-teacher to shape token-level advantages according to trajectory correctness. Unlike prior self-distilled RLVR methods, CAST does not require reference-solution-conditioned teacher scoring, keeps the self-teacher log-probability gap active throughout training, and applies bidirectional local advantage sign reversal: teacher-negative tokens in correct trajectories can receive negative token-level advantages, while teacher-positive tokens in incorrect trajectories can receive bounded positive local advantages. For zero-variance all-correct and all-wrong groups, CAST assigns bounded sign-constrained base advantages, so these otherwise zero-gradient groups can contribute verifier-signed token feedback. Experiments on mathematical reasoning show that CAST improves RLVR training while retaining a lightweight, verifier-grounded trajectory-level objective.
| Comments: | 10 pages |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.00172 [cs.AI] |
| (or arXiv:2606.00172v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.00172 arXiv-issued DOI via DataCite |
Submission history
From: Yang Li [view email]
[v1]
Fri, 29 May 2026 13:21:30 UTC (446 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.