Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation
Quick Answer
This paper shows that The On-Policy Diffusion Language Model (OPDLM) transforms autoregressive models into diffusion models using On-Policy Distillation, achieving 15x to 7,000x fewer training tokens while maintaining strong performance across various tasks.
Quick Take
This method addresses distribution shifts and train-inference mismatches inherent in standard DLMs, enhancing knowledge retention from the original ARLM.
Key Points
- OPDLM uses self-OPD for efficient ARLM-to-DLM transformation.
- It reduces training token requirements significantly, from 15x to 7,000x.
- The model eliminates train-inference mismatches found in standard DLMs.
- Knowledge retention from the original ARLM is enhanced through distillation.
- Empirical results show strong performance across a variety of tasks.
Paper Resources
Source Excerpt
arXiv:2606. 06712v1 Announce Type: new Abstract: We study the transformation of autoregressive models (ARLMs) into diffusion language models (DLMs). Rather than pretraining from scratch, prior work replaces the causal attention in ARLMs with bidirectional attention and then trains the resulting model using a DLM objective. However, these approaches incur two distribution shifts. First, transitioning from a next-token prediction objective to a DLM objective can discard knowledge acquired by the ARLM during training.
…
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.