VisualPatchWorld: Code World Models as Latent Structured Representations for Planning
Quick Answer
This paper shows that VisualPatchWorld (VPW) introduces a novel approach to world modeling by representing dynamics as code, achieving a 69.0% mean planning success rate, outperforming previous models by 23.5 points.
Quick Take
This method allows for inspection and modification of models, making it a practical solution for planning tasks in robotics.
Key Points
- VPW selects qualitative dynamics using short active probes for effective modeling.
- The model fits parameters by minimizing multi-step prediction error from state-action traces.
- It achieves a 69.0% mean planning success, surpassing previous code-based models.
- The approach is particularly effective when selecting the correct qualitative dynamics.
- Code for VPW is publicly available for further research and application.
DeepSignal Analysis
What happened
VisualPatchWorld (VPW) presents a new method for world modeling by encoding dynamics as code. This approach achieved a mean planning success rate of 69.0%, surpassing previous models by 23.5 points. VPW allows for model inspection and modification, making it applicable for robotics planning tasks.
Key evidence
- VPW represents world dynamics as code, selecting qualitative forms and fitting parameters from state-action traces to minimize prediction error.
- The method achieved a 69.0% mean planning success rate, outperforming the strongest prior code-based model by 23.5 points.
- The largest improvements in performance occur when selecting the correct qualitative dynamics, particularly in navigation and grasp-rich control tasks.
Why it matters
The development of VPW signifies a shift towards practical world modeling in robotics, combining the benefits of neural predictors and physics engines. By enabling model inspection and modification, VPW could enhance the adaptability and effectiveness of robotic planning systems. This advancement may lead to more efficient and reliable robotic applications in dynamic environments.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:Different research lines use the term world model in different ways, yet they share a common aim: to capture how the world evolves under action in a form that supports perception, simulation, and planning. Two prominent realizations are neural predictors that learn dynamics in continuous vector spaces, and hand-built physics engines that expose explicit state and physical laws. Neural predictors scale from data but leave the form of the dynamics implicit; physics engines are inspectable and editable but difficult to construct at scale. We introduce VisualPatchWorld (VPW), which represents world dynamics as code. VPW first selects a qualitative dynamical form with short active probes, then fits that form's free parameters from recorded state-action traces by minimizing multi-step prediction error. The resulting programs can be rolled forward like a simulator, inspected in source form, and used inside model-predictive control; image-derived scene graphs can supply the live state at replan time. Across comparisons with prior code-based world models, VPW attains 69.0% mean planning success and exceeds the strongest code baseline by 23.5 points. The largest gains arise when choosing the correct qualitative dynamics is essential. Under the same planner, the induced models approach ground-truth engine success on navigation and grasp-rich control; a residual gap remains for contact-rich pushing, and checking a shortlist of promising plans in the engine closes most of that gap. These results establish a practical route toward automatically constructed code world models that are useful for planning. Code is available at this https URL.
| Subjects: | Computation and Language (cs.CL); Robotics (cs.RO) |
| Cite as: | arXiv:2607.25236 [cs.CL] |
| (or arXiv:2607.25236v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.25236 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiaxin Bai [view email]
[v1]
Tue, 28 Jul 2026 03:23:47 UTC (1,025 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.