Better Starts, Better Ends: Bootstrapped Iterative Self-Reasoning Distillation for Compressed Reasoning
Quick Answer
The BIRD method enhances reasoning efficiency by improving model rollouts before training, achieving a MATH-500 accuracy increase from 86.2% to 92.0% while reducing response length from 3,099 to 1,115 tokens in Qwen3-8B.
Quick Take
This two-stage self-reasoning distillation addresses initialization bottlenecks in existing models, leading to better performance on benchmarks.
Key Points
- BIRD method samples concise solutions to enhance reasoning efficiency.
- Achieves MATH-500 accuracy improvement from 86.2% to 92.0% on Qwen3-8B.
- Reduces average response length from 3,099 to 1,115 tokens.
- Addresses initialization bottlenecks in existing self-distillation methods.
- Demonstrates prefix support as crucial for efficient reasoning distillation.
Paper Resources
Source Excerpt
Large reasoning models often solve problems through long chain-of-thought (CoT) traces, yet much of this computation is spent on redundant derivations, repeated self-verification, and detours that do not improve the final answer. Existing on-policy self-distillation methods reduce this cost by matching a student model to a concise copy of itself on prefixes sampled from the student's own rollouts. We show that this objective has an initialization bottleneck. Since supervision is applied only to
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.