Self-Generated Error Training for Token Editing in Diffusion Language Models
Quick Answer
This paper shows that The self-generated T2T editing method enhances LLaDA2.1's performance by addressing training-inference mismatches, improving accuracy while reducing edit intensity.
Quick Take
This approach involves a no-gradient draft pass and a recovery supervision pass, leading to fewer transcription errors and excessive self-corrections in generated outputs.
Key Points
- Introduces self-generated T2T editing for LLaDA2.1, improving accuracy.
- Addresses training-inference mismatch by using model-generated corruptions.
- Reduces T2T edit intensity, minimizing final-digit transcription errors.
- Implemented as a short LoRA continued-pretraining pass.
- Evaluated on multiple benchmarks with unchanged inference parameters.
Paper Resources
Source Excerpt
arXiv:2606. 17175v1 Announce Type: new Abstract: Token-to-token (T2T) editing lets LLaDA2. 1 revise committed tokens during block-diffusion decoding. The released recipe trains this editor on random vocabulary corruptions, but at inference the editor sees the model's own fluent, high-confidence draft errors instead.
We study this training-inference mismatch and propose self-generated T2T, which performs a no-gradient draft pass, fills masked positions with predicted tokens, and supervises recovery in a second pass under these self-generated corruptions. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.