LoRi: Low-Rank Distillation for Implicit Reasoning
Quick Answer
LoRi introduces a low-rank distillation framework for implicit reasoning in large language models like LLaMA and Qwen, enhancing performance on multi-step tasks.
Quick Take
The method aligns reasoning trajectories in a low-rank tensor subspace, achieving results close to explicit chain-of-thought prompting and outperforming previous iCoT methods across various benchmarks.
Key Points
- LoRi aligns teacher and student reasoning trajectories in a shared low-rank tensor subspace.
- The framework captures global reasoning structure while enabling a compact latent process.
- Evaluated on LLaMA and Qwen, it shows consistent improvement on mathematical reasoning tasks.
- Performance approaches explicit chain-of-thought accuracy, especially on challenging multi-step tasks.
- Outperforms previous implicit chain-of-thought distillation methods across multiple model families.
Paper Resources
Source Excerpt
Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in , but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the glob
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →RF-Agent: A Practical Framework for Building Language Agents for RFIC Design
RF-Agent introduces a novel framework for RF circuit design using , creating a unique RF-domain reasoning dataset with over 11,000 samples. The study reveals that domain-specific supervised fine-tuning and semantic retrieval strategies significantly enhance RF reasoning performance, particularly for smaller models.