Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
Quick Answer
This paper shows that An automated pipeline generates VQA datasets from private radiology reports and 3D oncology images, enabling evaluation of vision-language models (VLMs) without human annotation.
Quick Take
An automated pipeline generates VQA datasets from private radiology reports and 3D oncology images, enabling evaluation of (VLMs) without human annotation. Six VLMs showed no dominant performance, highlighting dataset-specific visual reliance, particularly in liver and lung imaging. The open agent skill allows for in-house deployment.
Key Points
- Automated pipeline creates multiple-choice VQA datasets from radiology reports and 3D images.
- Zero-shot evaluation of six VLMs shows no dominant model across all tested scenarios.
- Visual reliance varies by dataset; liver questions require images, while lung CT can be solved without.
- The benchmark avoids human annotation, ensuring contamination-controlled evaluation.
- The pipeline is available as an open agent skill for further use.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 02809v1 Announce Type: new Abstract: Evaluating (VLMs) on medical images requires benchmarks that are clinically grounded, scalable, and controlled for evaluation confounds. Existing public benchmarks are limited in scale, manually annotated, or potentially leaked into VLM pretraining corpora.
We present an automated agent-driven pipeline that generates multiple-choice VQA datasets directly from paired private radiology reports and 3D oncology imaging, producing two complementary question types: RADS-style questions deterministically derived from clinician-defined reporting schemas, and radiology report-derived questions generated by an from radiologist findings and verified against the source report.
Applied to four in-house cancer cohorts, the pipeline yields an instance-contamination-controlled benchmark without per-question human annotation. Zero-shot evaluation of six VLMs reveals no dominant model and substantial headroom across all cells.
A blind ablation reveals that visual reliance is highly dataset-specific: liver Report-derived questions genuinely require the image, while Lung CT is essentially solvable without it - the leading closed model exceeds its sighted accuracy on Lung CT when blinded - indicating that even private clinical data does not guarantee a contamination-controlled read of visual capability. The pipeline is released as an open agent skill for in-house redeployment.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.