Automated Report-Derived Oncology VQA Benchmark for Evaluating Vision-Language Models on 3D Medical Imaging
Quick Take
An automated pipeline generates VQA datasets from private radiology reports and 3D oncology images, enabling evaluation of vision-language models (VLMs) without human annotation. Six VLMs showed no dominant performance, highlighting dataset-specific visual reliance, particularly in liver and lung imaging. The open agent skill allows for in-house deployment.
Key Points
- Automated pipeline creates multiple-choice VQA datasets from radiology reports and 3D images.
- Zero-shot evaluation of six VLMs shows no dominant model across all tested scenarios.
- Visual reliance varies by dataset; liver questions require images, while lung CT can be solved without.
- The benchmark avoids human annotation, ensuring contamination-controlled evaluation.
- The pipeline is available as an open agent skill for further use.
Article Content
From source RSS / original summaryarXiv:2606. 02809v1 Announce Type: new Abstract: Evaluating vision-language models (VLMs) on medical images requires benchmarks that are clinically grounded, scalable, and controlled for evaluation confounds. Existing public benchmarks are limited in scale, manually annotated, or potentially leaked into VLM pretraining corpora.
We present an automated agent-driven pipeline that generates multiple-choice VQA datasets directly from paired private radiology reports and 3D oncology imaging, producing two complementary question types: RADS-style questions deterministically derived from clinician-defined reporting schemas, and radiology report-derived questions generated by an LLM from radiologist findings and verified against the source report.
Applied to four in-house cancer cohorts, the pipeline yields an instance-contamination-controlled benchmark without per-question human annotation. Zero-shot evaluation of six VLMs reveals no dominant model and substantial headroom across all cells.
A blind ablation reveals that visual reliance is highly dataset-specific: liver Report-derived questions genuinely require the image, while Lung CT is essentially solvable without it - the leading closed model exceeds its sighted accuracy on Lung CT when blinded - indicating that even private clinical data does not guarantee a contamination-controlled read of visual capability. The pipeline is released as an open agent skill for in-house redeployment.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Plan2Map: A Multimodal Benchmark for Document-Grounded Geospatial Boundary Reconstruction from Planning Records
Plan2Map introduces a 208-case benchmark for reconstructing geospatial boundaries from UK planning documents. The GeoPlanAgent system achieves a mean IoU of 0.736, significantly outperforming baseline models, highlighting the challenges in localization and map registration.
