Distilling LLM Feedback for Lean Theorem Proving
Quick Answer
The paper introduces Feedback Distillation, enhancing Lean4 theorem-proving by enabling token-level supervision from language model feedback.
Quick Take
This method improves diversity in generated trajectories over , leading to higher policy entropy and better pass@k scaling, suggesting a promising approach for complex reasoning post-training.
Key Points
- Feedback Distillation provides token-level supervision using privileged feedback from language models.
- The method maintains greater trajectory diversity than GRPO, enhancing policy entropy.
- Combining GRPO with Feedback Distillation initialization yields superior performance.
- Evaluation focused on Lean4 theorem-proving showcases improved pass@k scaling.
- Addresses challenges of sparse rewards and limited exploration in existing algorithms.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 30861v1 Announce Type: new Abstract: Post-training for reasoning models typically combines supervised fine-tuning with reinforcement learning from verifiable rewards, most commonly with . However, this algorithm suffers from sparse rewards, limited exploration, and mode collapse.
Building upon recent works on self-distillation, we propose Feedback Distillation, a training method where the model is trained to match, at the token level, its own distribution conditioned on privileged feedback produced by a language model. Feedback Distillation offers token-level supervision and can inject external knowledge.
Evaluating our method for Lean4 theorem-proving, we find that Feedback Distillation maintains greater diversity in generated trajectories than GRPO, yielding higher policy entropy and better pass@k scaling. The two methods are complementary: initializing GRPO from a Feedback Distillation checkpoint outperforms either method alone. All in all, our results suggest a promising avenue to improve post-training for complex reasoning.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.