It Takes 8 Tokens: Weak-to-Strong Off-Policy RL via Auxiliary Branches
Quick Answer
The W2SPO method enhances off-policy reinforcement learning by integrating short auxiliary segments into target model trajectories, achieving a Pass@1 improvement from 62.3% to 64.2% and a 3.55x training speedup on mathematical reasoning tasks with 4B scale models.
Key Points
- W2SPO uses 8-token auxiliary segments to inform policy exploration.
- Achieves superior performance on mathematical reasoning benchmarks with 4B scale models.
- Outperforms post-trained baselines in terms of policy updates.
- Improves training speed by 3.55 times compared to vanilla .
- Addresses semantic redundancy in target model samples during reasoning tasks.
Paper Resources
Source Excerpt
Reinforcement learning with verifiable rewards has emerged as a standard approach for enhancing reasoning in , which typically optimizes the policy by contrasting multiple self generated rollouts. However, we identify a critical support limited bottleneck in this paradigm: on challenging reasoning tasks, the target model's samples often exhibit semantic redundancy, converging into the same erroneous "reasoning basins" that offer negligible reward contrast for policy updates.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.