IMCBench: A benchmark for multimodal LLMs in Image-grounded Medical Conversations
Quick Answer
IMCBench introduces a novel benchmark for multimodal large language models (LLMs) in medical conversations, pairing clinical images with synthetic patient profiles.
Quick Take
IMCBench introduces a novel benchmark for multimodal large language models (LLMs) in medical conversations, pairing clinical images with synthetic patient profiles. The evaluation of eight models, including Claude Opus 4.6, reveals that while it scores highest overall (3.61), safety concerns persist, particularly for malignant and rare conditions, highlighting the need for multi-dimensional assessment frameworks in medical AI.
Key Points
- IMCBench evaluates multimodal LLMs using real clinical images and synthetic patient profiles.
- Eight models benchmarked include Claude Opus 4.6, scoring highest at 3.61.
- Safety concerns were noted, especially for malignant and rare conditions, with a drop of -0.27.
- Ablation studies indicate both visual input and EHR context are crucial for safe guidance.
- Findings emphasize the need for multi-dimensional evaluation frameworks in medical AI.
Paper Resources
📖 Reader Mode
~2 min readAuthors:Maria Xenochristou, Ashutosh Joshi, Korosh Vatanparvar, Mohammad Abuzar Hashemi, Prasad Kasu, Deepak Bansal, Anchal Nema, Nivedita Wadhwa, Prashams S Jain, Rebecca Abraham, Will Kimbrough, Dilek Hakkani-Tur, Wilko Schulz-Mahlendorf
Abstract:Recent advances in large language models and vision-language models have enabled reasoning over multimodal data, offering opportunities for clinical applications such as decision support and triaging. However, existing medical AI benchmarks are fragmented: some support multi-turn dialogues but lack images, while others provide multimodal inputs but focus on single-turn QA tasks. To address this gap, we introduce IMCBench, an image-grounded, multi-turn medical conversation benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate realistic patient-clinician interactions. Each conversation is evaluated across three clinical dimensions: safety, accuracy, and appropriate use of uncertainty in diagnosis. We benchmark eight multimodal frontier models across four model families (Claude, GPT, Nova, and Llama), scoring each on a 1-5 scale using LLM-as-Jury scoring calibrated against expert clinician annotations. Our results show that Claude Opus 4.6 achieves the highest overall score (3.61), followed by Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29), though no model dominates all dimensions and safety degrades for both malignant and rare conditions ($\Delta$ = -0.27 each). Ablation studies further reveal that both visual input and EHR context contribute to safe guidance (safety drops of 0.18 and 0.23 on average when each is removed), with stronger models leveraging visual features more effectively. Together, these findings demonstrate that accurate clinical description does not guarantee safe patient guidance, motivating the need for multi-dimensional evaluation frameworks in medical AI.
| Comments: | Accepted at ECML PKDD 2026. 22 pages, 2 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2606.28556 [cs.AI] |
| (or arXiv:2606.28556v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2606.28556 arXiv-issued DOI via DataCite |
Submission history
From: Maria Xenochristou [view email]
[v1]
Fri, 26 Jun 2026 19:18:16 UTC (866 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Verification Horizon: No Silver Bullet for Coding Agent Rewards
As coding agents evolve, verifying solutions becomes more challenging than generating them, necessitating a focus on scalable, faithful, and robust verification methods. The study reveals that no fixed reward function can sustain effectiveness as model capabilities advance, emphasizing the need for verification to evolve alongside solution generation.