Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
Quick Answer
This paper shows that Masked Diffusion Language Models (MDLMs) outperform autoregressive models in text-based world modeling, achieving up to 47% absolute gains in zero-shot transfer across diverse environments.
Quick Take
The study introduces a training framework and provides 239,403 state-action trajectories, demonstrating improved coherence and groundedness without environment-specific fine-tuning.
Key Points
- MDLMs achieve better coherence and groundedness than AR models with 4x the parameters.
- Introduced a GRPO training framework with deterministic state checks for enhanced performance.
- 239,403 grounded state-action trajectories were curated across nine open-source environments.
- Zero-shot transfer ablations showed up to 47% gains without environment-specific fine-tuning.
- Behavioral analysis revealed failure modes and human evaluation on realism and utility.
DeepSignal Analysis
What happened
The study introduces Masked Diffusion Language Models (MDLMs) as a new approach to text-based world modeling, demonstrating significant performance improvements over autoregressive models. It presents a GRPO training framework and a dataset of 239,403 state-action trajectories, achieving up to 47% gains in zero-shot transfer across various environments without specific fine-tuning.
Key evidence
- MDLMs outperform autoregressive models in text-based world modeling, achieving up to 47% absolute gains in zero-shot transfer across diverse environments.
- The study curated 239,403 grounded state-action trajectories spanning nine open-source environments and twelve frontier model families.
- MDLMs provide better coherence, groundedness, and rollout diversity than autoregressive models, despite being over four times their parameter size.
Why it matters
This research highlights the limitations of traditional autoregressive models in reinforcement learning and proposes MDLMs as a more effective alternative for creating diverse training environments. The ability to achieve significant performance gains without environment-specific fine-tuning could streamline the development of more robust AI agents, enhancing their adaptability and effectiveness in real-world applications.
Paper Resources
Source Excerpt
Recent growth in reinforcement learning (RL) has surfaced a need for diverse, specialized training environments. Hand-curated environments with fixed task and reward difficulties become ineffective signals as model performance improves, and sparse rewards over long horizons induce mode collapse on specific workflows or tool structures. World models that simulate environment states have matched pure rollout performance, making them promising for scaling diversity on-demand. However, autoregressiv
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.