Supportive Token Revealing for Fast Diffusion Language Model Decoding
Quick Take
The AXON module enhances discrete diffusion language models by optimizing the quality-latency trade-off during decoding. It selectively reveals confident tokens to support uncertain ones, improving performance on reasoning and code-generation benchmarks while reducing function evaluations. This approach maintains or enhances accuracy across multiple models.
Key Points
- AXON is a training-free module for parallel decoding in diffusion language models.
- It shifts focus from safe token reveals to those that support later denoising.
- Experiments show improved quality-latency trade-off on reasoning and code-generation tasks.
- AXON reduces function evaluations while maintaining or improving accuracy.
- Applicable to existing parallel decoding strategies without replacing the base decoder.
Article Content
From source RSS / original summaryarXiv:2606. 04236v1 Announce Type: new Abstract: Discrete diffusion language models can generate text efficiently by updating multiple masked positions in parallel, but this parallelism introduces a quality-latency trade-off. Aggressive decoding may commit mutually dependent tokens too early, while conservative decoding requires many denoising steps. Existing methods address this tension by deciding which tokens are safe to reveal using confidence or dependency criteria.
However, avoiding unsafe commits does not necessarily make the remaining masked sequence easy to decode, since uncertain tokens may depend on masked tokens, creating a bottleneck for denoising steps. We propose AXON, a training-free module that can be added on top of existing parallel decoding strategies for diffusion language models. Rather than replacing the base decoder, AXON monitors the remaining uncertain masked tokens and intervenes only when their current state suggests that additional context is needed.
It then shifts the criterion from which tokens are safest to reveal to which confident reveals would best support later denoising. AXON selects anchors, confident masked tokens that uncertain positions attend to, using attention, uncertainty, and confidence signals.
Experiments on reasoning and code-generation benchmarks across multiple diffusion language models show that AXON improves the quality-latency trade-off of existing parallel decoders, often reducing the number of function evaluations while maintaining or improving accuracy.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.