Distilling LLM Feedback for Lean Theorem Proving
Quick Take
The paper introduces Feedback Distillation, enhancing Lean4 theorem-proving by enabling token-level supervision from language model feedback. This method improves diversity in generated trajectories over GRPO, leading to higher policy entropy and better pass@k scaling, suggesting a promising approach for complex reasoning post-training.
Key Points
- Feedback Distillation provides token-level supervision using privileged feedback from language models.
- The method maintains greater trajectory diversity than GRPO, enhancing policy entropy.
- Combining GRPO with Feedback Distillation initialization yields superior performance.
- Evaluation focused on Lean4 theorem-proving showcases improved pass@k scaling.
- Addresses challenges of sparse rewards and limited exploration in existing algorithms.
Article Excerpt
From source RSS / original summaryarXiv:2605. 30861v1 Announce Type: new Abstract: Post-training for reasoning models typically combines supervised fine-tuning with reinforcement learning from verifiable rewards, most commonly with GRPO. However, this algorithm suffers from sparse rewards, limited exploration, and mode collapse.
Building upon recent works on self-distillation, we propose Feedback Distillation, a training method where the model is trained to match, at the token level, its own distribution conditioned on privileged feedback produced by a language model. Feedback Distillation offers token-level supervision and can inject external knowledge.
Evaluating our method for Lean4 theorem-proving, we find that Feedback Distillation maintains greater diversity in generated trajectories than GRPO, yielding higher policy entropy and better pass@k scaling. The two methods are complementary: initializing GRPO from a Feedback Distillation checkpoint outperforms either method alone. All in all, our results suggest a promising avenue to improve post-training for complex reasoning.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.