Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
Quick Answer
This paper shows that The Strategy-Guided Policy Optimization (SGPO) method enhances reasoning in language models by replacing trajectory imitation with reusable strategy distillation, achieving a 2.2-point improvement on Qwen2.5-7B-Instruct over traditional methods.
Quick Take
The Strategy-Guided Policy Optimization (SGPO) method enhances reasoning in language models by replacing trajectory imitation with reusable strategy distillation, achieving a 2.2-point improvement on Qwen2.5-7B-Instruct over traditional methods. This approach allows models to better generalize problem-solving skills across novel tasks.
Key Points
- SGPO outperforms SFT, on-policy RL, and hybrid-policy baselines in experiments.
- A token-level forward-KL objective selectively transfers strategy conditioning into unguided policies.
- Adaptive instance-level weighting enhances guidance based on model competence.
- Experiments conducted on four mathematical benchmarks across two model families.
- SGPO improves average scores significantly, demonstrating better generalization capabilities.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Distilling reasoning capabilities from strong to weak language models typically involves imitating specific solution trajectories, effectively transferring what to answer rather than how to reason. This trajectory-level imitation encourages memorization of instance-specific steps rather than acquisition of transferable problem-solving skills, limiting generalization to novel problems. We propose Strategy-Guided Policy Optimization (SGPO), which replaces instance-level trajectory imitation with reusable strategy distillation. SGPO extracts structured strategy descriptions from strong-model responses and, for each problem, constructs both autonomous and strategy-guided trajectories to enable direct comparison of the model's behavior with and without strategic guidance. The framework then addresses two key questions. For how to distill, a token-level forward-KL objective selectively transfers the distributional shift induced by strategy conditioning into the unguided policy, with proximal constraints ensuring stability. For when to distill, adaptive instance-level weighting strengthens guidance when autonomous exploration falls short and reduces it as the model's own competence grows. Experiments on four mathematical benchmarks across two model families show that SGPO consistently outperforms SFT, on-policy RL, and hybrid-policy baselines, improving the average score by 2.2 points over the strongest baseline on Qwen2.5-7B-Instruct. Analysis reveals that the forward-KL objective provides an inherently selective distillation signal that outperforms direct trajectory imitation, and that strategy distillation exhibits complementary scaling with base model capability.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.24064 [cs.AI] |
| (or arXiv:2606.24064v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.24064 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tianyuan Shi [view email]
[v1]
Tue, 23 Jun 2026 02:14:12 UTC (378 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Arbor: Tree Search as a Cognition Layer for Autonomous Agents
Arbor introduces a framework utilizing structured tree search for optimizing LLM inference, achieving up to 193% throughput-latency improvement compared to vendor-optimized systems. It employs an Orchestrator and Critic agent for stability and coordination, demonstrating hardware-agnostic performance with minimal variance.