MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
Quick Answer
MolmoMotion introduces a novel approach to 3D point motion forecasting using language instructions, achieving significant improvements over existing baselines on the PointMotionBench.
Quick Take
The model, trained on 1.16M videos, accurately predicts diverse motion patterns and enhances robot manipulation training efficiency.
Key Points
- MolmoMotion-1M corpus includes 1.16M videos with annotated 3D point trajectories.
- PointMotionBench features 111 object categories and 61 motion types for benchmarking.
- MolmoMotion model supports autoregressive prediction and flow-matching trajectory generation.
- The model significantly outperforms existing motion prediction methods on PointMotionBench.
- Learned 3D motion prior enhances training efficiency for robot manipulation tasks.
Paper Resources
Source Excerpt
Motion forecasting is central to visual intelligence: agents must anticipate how objects will move in order to plan actions, reason about physical interactions, and synthesize realistic futures. We argue that 3D points in world coordinates provide a general representation that is class-agnostic, view-stable, compact, and directly useful for downstream tasks. We formalize the task of goal-conditioned 3D point motion forecasting: given a short visual history, a set of 3D query points on an object
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.