3D Primitives are a Spatial Language for VLMs
arXiv cs.CV·Junze Liu, Kun Qian, Florian Dubost, Kai Zhong, Arvind Srinivasan, Nan Chen, Anping Wang, Sam Zhang, Alejandro Mottini, Qingjun Cui, Tian Wang
3d agoQuick Take
3D geometric primitives enhance spatial reasoning in vision-language models through innovative benchmarks and techniques.
Key Points
- Introduces SpatialBabel benchmark for evaluating VLMs on 3D reconstruction.
- Code-CoT improves spatial reasoning without training.
- S$^{3}$-FT enhances visual reasoning using self-supervised learning.
📖 Reader Mode
~2 min readAuthors:Junze Liu, Kun Qian, Florian Dubost, Kai Zhong, Arvind Srinivasan, Nan Chen, Anping Wang, Sam Zhang, Alejandro Mottini, Qingjun Cui, Tian Wang
Abstract:Vision-language models (VLMs) exhibit a striking paradox: they can generate executable code that reconstructs a 3D scene from geometric primitives with correct object counts, classes, and approximate positions, yet the same models fail at simpler spatial questions on the same image. We show that 3D geometric primitives (cubes, spheres, cylinders, expressed in executable code) serve as a powerful intermediate representation for spatial understanding, and exploit this through three contributions. First, we introduce \textbf{\textsc{SpatialBabel}}, a benchmark evaluating fourteen VLMs on primitive-based 3D scene reconstruction across six \emph{scene-code languages} (programming languages and declarative formats for 3D primitive scenes), revealing that a single model's object-detection F1 can vary by up to $5.7\times$ across languages. Second, we propose \textbf{Code-CoT} (Code Chain-of-Thought), a training-free inference strategy that routes spatial reasoning through primitive-based code generation. Code-CoT lifts the SpatialBabel-QA-Score by up to $+6.4$\% on primitive scenes and real-photo CV-Bench-3D accuracy by $+5.0$\% for VLMs with strong coding capabilities. Third, we propose \textbf{S$^{3}$-FT} (Self-Supervised Spatial Fine-Tuning), which self-supervisedly distills primitive spatial knowledge into general visual reasoning by parsing the model's own this http URL primitive-reconstructions into structured annotations and fine-tuning on the result, with \emph{no human labels and no teacher model}. Training on primitive images alone, S$^3$-FT improves Qwen3-VL-8B by $+4.6$ to $+8.6$\% on SpatialBabel-Primitive-QA, $+9.7$\% on CV-Bench-2D, and $+17$\% on HallusionBench; the recipe transfers across model families. These results establish geometric primitives in code as both a diagnostic and a transferable spatial vocabulary for VLMs. We will release all artifacts upon publication.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Databases (cs.DB) |
| Cite as: | arXiv:2605.12586 [cs.CV] |
| (or arXiv:2605.12586v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.12586 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Junze Liu [view email]
[v1]
Tue, 12 May 2026 17:57:21 UTC (3,953 KB)
— Originally published at arxiv.org
More from arXiv cs.CV
See more →2d ago
FeaturedOriginal
CoReDiT: Spatial Coherence-Guided Token Pruning and Reconstruction for Efficient Diffusion Transformers
AI Summary
CoReDiT enhances Diffusion Transformers by optimizing token pruning for efficiency and quality.