Read the Trace, Steer the Path: Trajectory-Aware Reinforcement Learning for Diffusion Language Models
Quick Answer
This paper shows that CAPR (Cached-Amortized Path Refinement) enhances reinforcement learning for diffusion language models (dLLMs) by summarizing denoising traces into compact path states.
Quick Take
It achieves a new state of the art in RL-tuned dLLMs, outperforming tree-structured baselines on benchmarks like Sudoku with reduced compute costs, achieving 0.75x the cost of flat rollouts and 0.6x of tree rollouts.
Key Points
- CAPR reduces rollout generation costs to 0.75x of flat rollouts and 0.6x of tree rollouts.
- Achieves new state of the art for RL-tuned dLLMs on benchmarks like Sudoku and Math500.
- Utilizes cached trajectory states for efficient sibling continuation generation.
- Records path-state and block-progress features for improved local supervision.
- Matches tree-structured baseline performance with less than one third of the compute.
Paper Resources
Source Excerpt
From the original publisher, up to about 700 charactersarXiv:2606. 04396v1 Announce Type: new Abstract: Diffusion (dLLMs) generate responses by iteratively unmasking and revising many positions in parallel. This process leaves a rich denoising trace depicting which tokens become confident, which remain unstable, and when commitments form. Existing dLLM reinforcement learning methods use this signal only weakly. Flat rollouts are cheap, but assign a single outcome reward to the whole trajectory.
Tree rollouts provide finer, verifiable training signals by branching partial trajectories and propagating leaf rewards upward, but are compute intensive. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.