MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models
Quick Answer
The MMBU benchmark introduces the largest biomedical vision-language evaluation, covering 35 submodalities and assessing 17 VLMs.
Quick Take
The MMBU benchmark introduces the largest biomedical vision-language evaluation, covering 35 submodalities and assessing 17 VLMs. It reveals that while medical adaptation improves some models, high accuracy on existing benchmarks may obscure visual perception issues.
Key Points
- MMBU is the largest biomedical vision-language benchmark to date.
- It evaluates 15 open-weight and 2 frontier vision-language models.
- The benchmark includes ungrounded and grounded classification, and object detection.
- High accuracy on established benchmarks can mask deficiencies in visual perception.
- Robust visual perception is critical for diverse biomedical imaging workflows.
Article Content
From source RSS / original summaryarXiv:2606. 06696v1 Announce Type: new Abstract: Vision and language models (VLMs) hold immense promise to transform biomedical imaging workflows, from detecting lesions in chest X-rays to profiling cellular features in microscopy. Realizing this potential, however, requires robust and fine-grained visual perception. Models need to correctly interpret subtle features in images, and they must do so across diverse biomedical modalities, scales, and contexts. Nevertheless, current benchmarks remain limited.
To address these gaps, we introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark. It is the largest biomedical vision and language benchmark to date, covering 35 submodalities with rich structured metadata. It includes both open and closed versions of ungrounded classification, grounded classification, and object detection, enabling systematic evaluation of model performance across biological scales, clinical settings, and imaging modalities.
Evaluating 15 open-weight and 2 frontier VLMs, we find that while medical adaptation provides measurable gains for some models, the high accuracy often reported on established benchmarks can mask deficiencies in visual perception and domain generalization.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.