Diffusion Transformer World-Action Model for AV Scene Prediction
Quick Answer
This paper shows that The Diffusion Transformer (DiT) model predicts future scenes for autonomous vehicles with 40% lower steering RMSE compared to traditional encoders, achieving KID scores of 0.078 versus 0.375 for regression, demonstrating superior action-controllability and practical deployment without ground truth.
Key Points
- DiT uses a latent world model to predict scenes based on ego-actions.
- Achieves 40% lower steering RMSE compared to the best single-frame encoder.
- Diffusion model shows KID score of 0.078, outperforming regression by 4.8x.
- Action-controllability is validated with Spearman correlation of 0.81.
- Compact 1.7M-parameter model recovers full ground-truth motion magnitude.
Paper Resources
Source Excerpt
arXiv:2606. 12987v1 Announce Type: new Abstract: Action-conditioned world models let an autonomous vehicle predict future camera scenes from its own planned controls, enabling planning and simulation without real-world rollouts, but at compact, trainable scale the futures are ambiguous and the field's standard distortion metrics actively mislead: they reward a blurry regression mean over a realistic prediction.
We confront this with a compact latent world model that, given the present front-camera latent and a sequence of ego-actions, predicts future scene latents a frozen decoder renders to $256 \times 256$ frames up to 8 seconds ahead, evaluated on 150 held-out nuScenes scenes. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.