Neural Voxel Dynamics: Learning Implicit 3D Physics via Volumetric Feature Advection
Quick Answer
The proposed self-supervised framework learns implicit 3D physics from video signals using a Volumetric Latent Space, achieving high structural stability and physical plausibility on benchmarks like CLEVERER and PhysInOne, without relying on traditional physics engines.
Key Points
- Introduces Volumetric Feature Advection for learning 3D physics from videos.
- Achieves good performance on CLEVERER, PhysInOne, and PhysGaia benchmarks.
- No reliance on physics engine states or labels during training.
- Tracks material states implicitly within high-dimensional V-JEPA features.
- Enables simulation of complex phenomena like rigid body motion in fluid.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We present a self-supervised framework for learning implicit 3D physical dynamics directly from video-derived supervisory signals. While current generative video models achieve high visual fidelity, they lack a 3D geometric foundation, often resulting in physical inconsistencies and a failure to maintain object permanence. We address this by shifting the predictive bottleneck from 2D image space to a `lifted' 3D Volumetric Latent Space. Our method unprojects semantic features from a Video Joint-Embedding Predictive Architecture (V-JEPA) into a voxelized grid, grounded by monocular depth priors. This lifting enables a Volumetric Feature Advection to learn an action-conditioned transition operator that treats physics as a spatio-temporal state advection problem, i.e., learn implicit 3D physics. Unlike state-of-the-art hybrid models that rely on explicit classical simulators for training and/or inference, our architecture tracks material states implicitly within high-dimensional V-JEPA features. This allows for the emergent simulation of heterogeneous phenomena (e.g., rigid body motion in fluid flow) within a single, unified pipeline. Supervised solely via end-to-end video-derived signal plus action conditions, without access to physics engine internal states, labels, or surrogate models, our model demonstrates good long-term structural stability and physical plausibility on multiple benchmarks (CLEVERER, PhysInOne, PhysGaia). We believe that this work opens a scalable pathway toward general-purpose dynamic world models that internalize the 3D invariants of the physical world solely through passive observation of monocular videos.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2606.26410 [cs.CV] |
| (or arXiv:2606.26410v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.26410 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Zican Wang [view email]
[v1]
Wed, 24 Jun 2026 22:06:35 UTC (27,859 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.