ReportMedSAM: Guiding Segmentation Through Radiology Reports
Quick Answer
ReportMedSAM introduces a report-driven framework that utilizes a learnable concept bank and a frozen medical vision-language encoder (BiomedCLIP) to enhance segmentation accuracy from free-form radiology reports.
Quick Take
ReportMedSAM introduces a report-driven framework that utilizes a learnable concept bank and a frozen medical vision-language encoder (BiomedCLIP) to enhance segmentation accuracy from free-form radiology reports. Evaluated on the AbdomenAtlas 3.0 dataset, it shows robust performance against clinical synonyms and allows for seamless integration of new concepts without retraining existing models.
Key Points
- Utilizes a learnable concept bank to replace traditional extraction methods.
- Achieves competitive segmentation accuracy on the AbdomenAtlas 3.0 dataset.
- Robust against variations in clinical terminology, such as 'renal' vs. 'kidney'.
- Supports dynamic activation of task-specific Mixture-of-Experts modules during inference.
- Allows for parameter-isolated extensions without retraining existing components.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Free-form radiology reports contain rich clinical descriptions, yet converting them for reliable segmentation remains challenging due to the inherent variability of natural language. Existing pipelines often rely on predefined organ phrases or brittle rule-based inference-time extraction, which limits their scalability to novel anatomical structures and makes them sensitive to linguistic variations. To address this, we propose ReportMedSAM, a report-driven framework that replaces discrete extraction with a learnable concept bank. By leveraging a frozen medical vision-language encoder (BiomedCLIP), we align organ-level concept embeddings with large-scale clinical corpora through contrastive learning, establishing mutually orthogonal semantic anchors. Our approach explicitly mitigates organ-level semantic collapse and ensures high robustness against diverse clinical synonyms (e.g., "renal" vs. "kidney" ). During inference, a clinical report is embedded and matched against this concept bank to dynamically activate task-specific Mixture-of-Experts (MoE) modules. This decoupled design allows new concepts and experts to be added without retraining existing components, providing a parameter-isolated extension mechanism while keeping previously learned experts unchanged. Evaluated on the AbdomenAtlas 3.0 dataset, ReportMedSAM effectively interprets free-form reports, achieves competitive segmentation accuracy, and demonstrates seamless, non-interfering extension to novel clinical tasks.
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.14116 [cs.CL] |
| (or arXiv:2607.14116v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.14116 arXiv-issued DOI via DataCite |
Submission history
From: Anghong Du [view email]
[v1]
Fri, 8 May 2026 13:32:15 UTC (2,937 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.