Diversity Over Frequency: Rethinking Tool Use in Visual Chain-of-Thought Agents
Quick Take
This paper reveals that visual agents experience a 'tool-use collapse' phenomenon, where increased tool usage does not correlate with improved reasoning performance in complex tasks like 3D spatial reasoning and medical VQA. By introducing an entropy regularization term, the authors enhance rollout diversity, leading to better outcomes despite reduced tool utilization.
Key Points
- Visual agents show a 'tool-use collapse' in complex reasoning tasks.
- Eliminating tool use degrades performance, while incentivizing it yields marginal gains.
- Entropy regularization improves rollout diversity and overall performance.
- Findings apply to both 3D spatial reasoning and medical visual question answering.
- Broader exploration enhances reasoning despite declining tool usage.
Article Content
From source RSS / original summaryarXiv:2606. 00096v1 Announce Type: new Abstract: Visual agents employ external visual tools within visual chains of thought to incorporate fine-grained evidence. While prior work has mainly studied these tools in visual search tasks, their role in more complex visual reasoning remains underexplored.
In this paper, we move beyond simple visual search tasks to investigate more challenging tasks, including 3D spatial reasoning and medical visual question answering, where agents must integrate tool-acquired local evidence with the global context. We identify a {tool-use collapse phenomenon: models progressively stop using tools while still achieving higher task accuracy.
Moreover, we observe a clear asymmetry: (i) completely eliminating tool use degrades performance, whereas (ii) incentivizing tool use yields only marginal gains despite substantially increasing usage. We find that vanilla training and tool-use encouragement both reduce rollout diversity, explaining why higher tool use does not yield stronger reasoning performance.
Motivated by these findings, we add an entropy regularization term to encourage diverse rollout exploration, achieving the best performance despite gradually declining tool usage. % We further observe similar dynamics on medical VQA, suggesting that tool-use collapse is not limited to 3D spatial reasoning. Overall, our findings suggest a training-time view of tools as scaffolding, where broader exploration over language generation and visual tool invocation improves reasoning despite tool-use collapse.
Project page: https://scaffolded-exploration. github. io
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, enabling efficient fine-tuning with only 0.11% parameter updates. It significantly enhances performance in few-shot learning and domain shifts across 15 biomedical imaging datasets, demonstrating robustness for clinical applications.