ReportMedSAM: Guiding Segmentation Through Radiology Reports
Quick Answer
ReportMedSAM introduces a report-driven framework that utilizes a learnable concept bank and a frozen medical vision-language encoder (BiomedCLIP) to enhance segmentation accuracy from free-form radiology reports.
Quick Take
Evaluated on the AbdomenAtlas 3.0 dataset, it shows robust performance against clinical synonyms and allows for seamless integration of new concepts without retraining existing models.
Key Points
- Utilizes a learnable concept bank to replace traditional extraction methods.
- Achieves competitive segmentation accuracy on the AbdomenAtlas 3.0 dataset.
- Robust against variations in clinical terminology, such as 'renal' vs. 'kidney'.
- Supports dynamic activation of task-specific Mixture-of-Experts modules during inference.
- Allows for parameter-isolated extensions without retraining existing components.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Free-form radiology reports contain rich clinical descriptions, yet converting them for reliable segmentation remains challenging due to the inherent variability of natural language. Existing pipelines often rely on predefined organ phrases or brittle rule-based inference-time extraction, which limits their scalability to novel anatomical structures and makes them sensitive to linguistic variations. To address this, we propose ReportMedSAM, a report-driven framework that replaces discrete extraction with a learnable concept bank. By leveraging a frozen medical vision-language encoder (BiomedCLIP), we align organ-level concept embeddings with large-scale clinical corpora through contrastive learning, establishing mutually orthogonal semantic anchors. Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms (e.g., "renal" vs. "kidney" ). During inference, a clinical report is embedded and matched against this concept bank to dynamically activate task-specific Mixture-of-Experts (MoE) modules. This decoupled design allows new concepts and experts to be added without retraining existing components, providing a parameter-isolated extension mechanism while keeping previously learned experts unchanged. Evaluated on the AbdomenAtlas 3.0 dataset, ReportMedSAM effectively interprets free-form reports, achieves competitive segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.14116 [cs.CL] |
| (or arXiv:2607.14116v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14116 arXiv-issued DOI via DataCite |
Submission history
From: Anghong Du [view email]
[v1]
Fri, 8 May 2026 13:32:15 UTC (2,937 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.