When Vision Speaks for Sound
arXiv cs.CV·Xiaofei Wen, Wenjie Jacky Mo, Xingyu Fu, Rui Cai, Tinghui Zhu, Wendi Li, Yanan Xie, Muhao Chen, Peng Qi
1d agoQuick Take
Video-capable MLLMs often misinterpret audio, relying on visual cues instead of verifying sound.
Key Points
- Models exhibit a Clever Hans effect in audio understanding.
- Thud framework introduces counterfactual audio edits for analysis.
- Best recipe improves performance by 28 percentage points.
📖 Reader Mode
~2 min readAbstract:Despite rapid progress in video-capable MLLMs, we find that their apparent audio understanding in videos is often vision-driven: models rely on visual cues to infer or hallucinate acoustic information, rather than verifying the audio stream. This issue appears across both state-of-the-art open-source omni models and leading closed-source models from providers such as Google and OpenAI. We characterize this failure mode as an audio-visual Clever Hans effect, in which models appear (falsely) audio-grounded, but actually exploit visual-acoustic correlations without verifying whether the audio and visual streams are truly aligned. To systematically study this behavior, we introduce Thud, an intervention-driven probing framework based on three counterfactual audio edits: Shift, which tests temporal synchronization; Mute, which tests sound existence; and Swap, which tests audio-visual consistency. Beyond diagnosis, we further study a two-stage alignment recipe: intervention-derived preference pairs teach audio verification, while event-level general video preferences regularize the model against over-specialization. Our best 10K-sample recipe improves average performance across the three intervention dimensions by 28 percentage points, while slightly improving performance on general video and audio-visual QA benchmarks.
| Comments: | 24 pages, 10 figures |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Sound (cs.SD) |
| Cite as: | arXiv:2605.16403 [cs.CV] |
| (or arXiv:2605.16403v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16403 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xiaofei Wen [view email]
[v1]
Wed, 13 May 2026 05:00:19 UTC (13,908 KB)
— Originally published at arxiv.org
More from arXiv cs.CV
See more →arXiv cs.CV·Jinhao Jing, Zheng Ma, Jinwei Liang, Qiannian Zhao, Shawn Chen, Jing Yang, Por Lip Yee, Prayag Tiwari, Jingjing Bai, Benyou Wang, Lewei Lu, Zhan Su
1d ago FeaturedOriginal
GeoSym127K: Scalable Symbolically-verifiable Synthesis for Multimodal Geometric Reasoning
AI Summary
GeoSym127K introduces a scalable neuro-symbolic framework for enhanced geometric reasoning in multimodal models.