RealWeather: Realistic and Scene-Faithful Weather Translation with Driving World Models
Quick Answer
RealWeather introduces a driving world model for realistic weather translation, leveraging Progressive Realism Bootstrapping to refine data from pseudo-style to authentic videos.
Quick Take
It outperforms existing methods in visual realism and structural integrity, enabling robust generation of long-tail weather scenarios and strong zero-shot generalization.
Key Points
- Employs Progressive Realism Bootstrapping for iterative data refinement.
- Utilizes a Pseudo-Clear Generation pipeline to enhance training realism.
- Introduces Scene-Fidelity RL Optimization to maintain structural integrity.
- Demonstrates superior performance in visual realism and structural preservation.
- Enables robust generation of long-tail weather scenarios.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Realistic weather translation is valuable for developing and evaluating autonomous driving systems, yet collecting paired videos of the same scenes under different weather conditions at scale is impractical. Existing methods therefore rely on synthetic data, 3D weather editing, or geometry-conditioned generation, often compromising weather realism or scene fidelity. We propose RealWeather, a driving world model for both realistic and scene-faithful weather translation. Our key idea is to learn authentic weather dynamics directly from real-world videos. Specifically, RealWeather employs Progressive Realism Bootstrapping, an iterative data-refinement strategy. Assisted by an auxiliary Pseudo-Clear Generation pipeline, training initially starts with pseudo-style conditioning videos. As training proceeds, these inputs are progressively replaced with increasingly realistic videos generated by the model itself. This strategy bridges the pseudo-to-real domain gap, allowing the model to adapt seamlessly to real-world input distributions and naturally support bidirectional clear adverse translation. Furthermore, to strictly enforce structural integrity and suppress hallucinations, we introduce Scene-Fidelity RL Optimization, a reward-driven policy optimization strategy that explicitly penalizes alterations to safety-critical driving elements. Extensive experiments demonstrate that RealWeather significantly outperforms existing methods in visual realism and structural preservation, while enabling robust long-tail weather scenario generation and strong zero-shot out-of-distribution generalization.
| Comments: | Under submission |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2608.02953 [cs.CV] |
| (or arXiv:2608.02953v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2608.02953 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Guanbin Li [view email]
[v1]
Mon, 3 Aug 2026 23:33:49 UTC (8,301 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.