Dive into the Scene: Breaking the Perceptual Bottleneck in Vision-Language Decision Making via Focus Plan Generation
Quick Answer
SceneDiver introduces a coarse-to-fine focus plan generation method for Vision-Language Models (VLMs) and Vision-Language-Action Models (VLAs), significantly reducing visual hallucinations and improving task execution efficiency.
Quick Take
Evaluations on standard benchmarks demonstrate enhanced performance in robotic manipulation and navigation tasks while maintaining computational efficiency.
Key Points
- SceneDiver constructs a holistic scene graph for initial scene comprehension.
- It decomposes tasks into simpler sub-problems through iterative recognition and analysis.
- The method reduces visual hallucinations for both and VLAs.
- A lightweight adapter distills focus ability for reactive control in VLAs.
- Code and data are available at the project's GitHub page.
Paper Resources
Source Excerpt
From the original publisher, up to about 700 charactersIn embodied vision-language decision making tasks such as robotic manipulation and navigation, Vision-Language and Vision-Language-Action Models ( & VLAs) are powerful tools with different benefits: VLMs are better at long-term planning, while VLAs are better at reactive control. However, their performance is limited by the same perceptual bottleneck: visual hallucinations arise due to the models' inability to distinguish task-relevant objects from distractors. In principle, accurate identif
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.