Factorization-Error-Free Discrete Diffusion Language Model via Speculative Decoding
Quick Answer
The FeF-DLLM model enhances discrete diffusion language modeling by eliminating factorization errors through prefix-conditioned token prediction, achieving a 5.04% accuracy boost and 3.86x inference speedup on benchmarks like GSM8K and MATH.
Key Points
- FeF-DLLM replaces independent token prediction with exact prefix-conditioned factorization.
- The model maintains token dependencies, improving generation efficiency.
- Achieves an average accuracy improvement of 5.04 percentage points.
- Inferences are sped up by an average of 3.86 times across various benchmarks.
- Proven to generate from the true joint distribution theoretically.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 14305v1 Announce Type: new Abstract: Discrete diffusion language models improve generation efficiency through parallel token prediction, but standard $X_0$ prediction methods introduce factorization errors by approximating the clean token posterior with independent token-wise distributions.
This paper proposes Factorization-Error-Free Discrete Diffusion Language Modeling (FeF-DLLM), which replaces independent clean-token prediction with an exact prefix-conditioned factorization of the clean posterior to better preserve token dependencies. To reduce the sequential cost introduced by prefix conditioning, FeF-DLLM further incorporates speculative decoding within diffusion denoising, accelerating inference while maintaining the parallel prediction and re-masking properties of DLLMs.
Theoretically, we prove that FeF-DLLM generates from the true joint distribution and derive its expected acceleration ratio. Experiments on GSM8K, MATH, HumanEval, and MBPP demonstrate that our method improves accuracy by an average of 5. 04 percentage points while achieving an average inference speedup of $3. 86\times$.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.