Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
Quick Answer
SceneDiver introduces a coarse-to-fine focus plan generation method for Vision-Language Models (VLMs) and Vision-Language-Action Models (VLAs), significantly reducing visual hallucinations and improving task execution efficiency.
Quick Take
Evaluations on standard benchmarks demonstrate enhanced performance in robotic manipulation and navigation tasks while maintaining computational efficiency.
Key Points
- SceneDiver constructs a holistic scene graph for initial scene comprehension.
- It decomposes tasks into simpler sub-problems through iterative recognition and analysis.
- The method reduces visual hallucinations for both and VLAs.
- A lightweight adapter distills focus ability for reactive control in VLAs.
- Code and data are available at the project's GitHub page.
Paper Resources
📖 Reader Mode
~2 min readAbstract:In embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models (VLMs & VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual hallucinations arise due to the models' inability to distinguish task-relevant objects from distractors. In principle, accurate identification and focus on critical objects while filtering out irrelevant ones is the key to break this limitation. A straightforward solution is one-step focus: directly attending to essential objects. However, this approach proves ineffective because effective focus inherently requires deep scene understanding. To this end, we propose SceneDiver, a coarse-to-fine focus plan generation method for VLMs leveraging their long-term planning abilities, that first constructs a holistic scene graph to establish initial comprehension, then progressively decomposes the task into simpler sub-problems through an iterative cycle of recognition, understanding, and analysis. To enable reactive control, we also design a lightweight adapter for distilling the deliberate focus ability into VLAs. Evaluations on standard embodied AI benchmarks confirm that our method substantially reduces visual hallucinations for both VLMs and VLAs, while preserving computational efficiency in tasks requiring fast execution. Our code and data are released at: this https URL.
| Comments: | Accepted at ICML 2026 |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Machine Learning (cs.LG); Robotics (cs.RO) |
| Cite as: | arXiv:2606.04046 [cs.CV] |
| (or arXiv:2606.04046v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.04046 arXiv-issued DOI via DataCite |
Submission history
From: Boyuan Xiao [view email]
[v1]
Tue, 2 Jun 2026 07:50:56 UTC (3,070 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.