CaRE Compute-aware Remasking Evaluation Protocol for Masked Diffusion Language Models
Quick Answer
This paper shows that The CaRE framework standardizes evaluation for masked diffusion language models (MDLMs) by controlling function evaluations, metrics, and stochasticity.
Quick Take
Findings show that temperature significantly affects MAUVE variance, and compute-matched comparisons alter previous strategy rankings. The protocol aims to clarify algorithmic improvements versus evaluation artifacts across various MDLM architectures.
Key Points
- CaRE audits MDLM remasking strategies using standardized function evaluations and multi-metric reporting.
- Temperature accounts for most MAUVE variance in evaluations of MDLMs.
- Compute-matched comparisons reversed several previously published strategy rankings.
- High-entropy remasking reduces MAUVE scores significantly at certain parameters.
- The evaluation protocol and leaderboard promote reproducibility in future MDLM research.
DeepSignal Analysis
What happened
The CaRE framework has been introduced to standardize the evaluation of masked diffusion language models (MDLMs). It addresses inconsistencies in previous evaluations by controlling for function evaluations, metrics, and stochasticity. The findings indicate that temperature significantly impacts MAUVE variance and that compute-matched comparisons can change the rankings of strategies previously published.
Key evidence
- The CaRE framework audits MDLM remasking strategies by standardizing the actual number of function evaluations (NFE) and enforcing multi-metric reporting.
- Temperature was found to explain the majority of MAUVE variance in the evaluations conducted using the CaRE framework.
- The CaRE leaderboard includes 12 open-weight MDLMs ranging from 150M to 8B parameters, indicating a broad application of the framework across different architectures.
Why it matters
The introduction of the CaRE framework is significant as it aims to clarify whether improvements in MDLMs are due to genuine algorithmic advancements or merely artifacts of evaluation conditions. This is crucial for the field, as it ensures that future claims about MDLM performance are reproducible and comparable. By addressing the inconsistencies in evaluations, researchers can better understand the true capabilities of these models.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Masked diffusion language models (MDLMs) are advancing rapidly, yet the evaluation standards needed to reliably interpret their progress have not kept pace. Despite MDLMs becoming competitive with autoregressive language models, seven recent remasking papers evaluate under incompatible settings, varying nominal step counts, metrics, and sampling temperatures without jointly controlling these factors, rendering their strategy rankings largely incomparable and leaving open whether reported gains reflect algorithmic improvements or evaluation artifacts. We present CaRE, a compute-aware evaluation framework that audits MDLM remasking strategies by standardizing actual number of function evaluations (NFE), enforcing multi-metric reporting, and explicitly controlling stochasticity. Applied to 7 remasking strategies across LLaDA-8B-Base and Dream-7B-Base at 4 stochasticity levels and 3 step budgets on OpenWebText and LM1B, CaRE reveals that: (i) temperature explains the majority of MAUVE variance, (ii) compute-matched comparisons reverse several published strategy rankings, and (iii) informed remasking and stochastic unmasking are in tension, with high-entropy remasking reducing MAUVE by 0.296 at 256 steps at unmask_temp=0.25 (p=0.020). A CaRE leaderboard covering 12 open-weight MDLMs (150M to 8B parameters) shows that this interaction direction holds across architectures and scales. These findings demonstrate that current MDLM evaluations can systematically conflate algorithmic improvements with hidden choices of compute and stochasticity. We release the evaluation protocol, implementation, and leaderboard to ensure future remasking claims are reproducible and comparable.
| Comments: | updated version |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.24763 [cs.AI] |
| (or arXiv:2607.24763v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.24763 arXiv-issued DOI via DataCite |
Submission history
From: Abhijit Chakraborty [view email]
[v1]
Thu, 4 Jun 2026 22:21:08 UTC (1,073 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.