Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark
Quick Answer
This paper shows that A new benchmark reveals that vision-language models (VLMs) often rely on textual priors rather than image content, with performance dropping significantly on harder variants.
Quick Take
Eleven models were tested, showing open-weight models degrade to a text-only accuracy of 1-9%. Techniques like post-training can help reduce this reliance.
Key Points
- A benchmark of 540 images across six reasoning categories was created.
- Open-weight models showed the largest performance drop on the hardest question variant.
- No-image ablation tests revealed a drop to 1-9% accuracy for open models.
- In-context exemplars improved accuracy significantly for .
- GRPO post-training yielded consistent performance gains across all test variants.
Paper Resources
Source Excerpt
arXiv:2606. 10400v1 Announce Type: new Abstract: (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.