Image2Sim: Scaling Embodied Navigation via Generative Neural Simulator
Quick Answer
Image2Sim is a real-time neural simulation framework that generates high-quality interactive environments from RGB-D image sequences, enabling scalable embodied navigation.
Quick Take
Image2Sim is a real-time neural simulation framework that generates high-quality interactive environments from RGB-D image sequences, enabling scalable embodied navigation. It creates nearly 20K interactive scenes and over 10 million navigation samples, significantly improving navigation models on major benchmarks and facilitating effective transfer to real-world scenarios.
Key Points
- Image2Sim decouples 3D spatial anchoring from photorealistic observation synthesis.
- It uses a feed-forward feature Gaussian model for efficient scene construction.
- The framework synthesizes over 10 million navigation training samples.
- Navigation models trained in Image2Sim show strong improvements on major benchmarks.
- Results indicate scalable neural simulation can enhance embodied navigation training.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Embodied navigation aims to build agents that interpret multimodal goals, reason in 3D space, and reach target destinations reliably in the real world. However, progress remains constrained by the lack of scalable, high-fidelity, and physically grounded interactive environments. Although real-world scanned datasets offer visual realism, they are limited by scale. In contrast, synthetic simulators scale more easily but often exhibit large sim-to-real gaps. We introduce Image2Sim, a real-time neural simulation framework that constructs high-quality interactive environments from posed RGB-D image sequences. The central idea is to decouple 3D spatial anchoring from photorealistic observation synthesis. For scene construction, Image2Sim uses a feed-forward feature Gaussian model that lifts posed RGB-D observations into a 3D feature-Gaussian representation in a single pass. For rendering, we propose a Geometry-Aware One-Step Pixel Flow model that transforms sparse and noisy Gaussian projections into high-quality panoramic RGB-D observations. Image2Sim also serves as a fully automated embodied data engine that generates high-fidelity observations, executable actions, and diverse navigation instructions at scale. It converts large collections of videos and images into nearly 20K interactive scenes and synthesizes more than 10 million navigation training samples. Navigation models trained entirely in these neural environments achieve strong improvements on major benchmarks and transfer effectively to real-world zero-shot settings. These results suggest that scalable neural simulation can serve as a practical training substrate for embodied navigation at scale.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Robotics (cs.RO) |
| Cite as: | arXiv:2607.05765 [cs.CV] |
| (or arXiv:2607.05765v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05765 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zihan Wang [view email]
[v1]
Tue, 7 Jul 2026 02:42:41 UTC (2,751 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.