DeSpatial: The devil of Agentic Spatial Reasoning Lies in 3D Itself
Quick Answer
DeSpatial introduces a novel approach to 3D spatial reasoning by decomposing it into perception and reasoning components, achieving leading performance on VSI-Bench and other benchmarks.
Quick Take
The model addresses inconsistencies in object identities and predictions, demonstrating that the bottleneck in spatial understanding lies in precise 3D perception itself.
Key Points
- DeSpatial reconciles independent perception outputs into a unified 3D scene.
- Achieves nearly full scores on five object-centric VSI-Bench tasks using ground-truth 3D boxes.
- Includes a joint segmentation-depth co-refinement and a cross-view instance fusion module.
- Demonstrates leading performance on VSI-Bench, MindCube-Tiny, and ViewSpatial.
- Reveals that the core challenge in spatial reasoning is rooted in 3D perception.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Yifei Liu, Xiaolong Zhou, Ziyang Gong, Yifan Zhou, Yuanyuan Gao, Yuning Gong, Yuchen Yang, Hongjie Zhang, Xue Yang, Xiao Sun, Zhihang Zhong
Abstract:Spatial agents have access to reconstruction and segmentation tools, yet inconsistent predictions and fragmented object identities can prevent their outputs from becoming reliable 3D scene information. A simple oracle experiment exposes this perception bottleneck: replacing tool-predicted perception with ground-truth 3D bounding boxes raises an agent to nearly full score across five object-centric VSI-Bench tasks, while retaining the same reasoning tools. Motivated by this finding, we believe the devil of spatial reasoning lies in 3D itself, and introduce DeSpatial(Devil of Spatial), which explicitly decomposes spatial understanding into perception and reasoning. DeSpatial has a shared 3D perception tool to reconcile independent perception outputs into a unified, metric, gravity-aligned 3D scene. This perception tool has two core components: a) a joint segmentation--depth co-refinement module that consolidates masks of the same object and removes projection artifacts caused by inconsistent segmentation and depth within a single frame; b) a cross-view instance fusion module that consolidate voxel-overlapping instances but not allowed to merge same-frame instances. This fusion module additionally fuses spatially disjoint fragments caused by insufficient multi-view observations, by checking their 3D structure and source mask appearance. After receiving the complete and unified 3D scene information, the agent chooses reasoning or computational tools to solve the spatial problem. DeSpatial achieves leading performance on VSI-Bench, MindCube-Tiny, and ViewSpatial, with additional long-video results on VSI-SUPER, revealing the bottleneck of spatial understanding lies in a precise 3D perception itself.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2610.20990 [cs.CV] |
| (or arXiv:2610.20990v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2610.20990 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yifei Liu [view email]
[v1]
Wed, 14 Oct 2026 22:05:23 UTC (6,265 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.