MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models
Quick Answer
MedSIGHT is a unified framework for medical large vision-language models (Med-LVLMs) that enhances visual comprehension by integrating pixel-level understanding.
Quick Take
MedSIGHT is a unified framework for medical large vision-language models (Med-LVLMs) that enhances visual comprehension by integrating pixel-level understanding. It employs a Region Perceiver module and a medical region codebook, achieving state-of-the-art performance in medical comprehension and segmentation tasks with only 72K multimodal instruction pairs.
Key Points
- Introduces Region Perceiver for region-centric token generation.
- Incorporates a medical region codebook into the LLM vocabulary.
- Achieves state-of-the-art results in diverse imaging modalities.
- Trained on 72K multimodal instruction pairs for effective learning.
- Combines progressive training strategy for stable module alignment.
Article Content
From source RSS / original summaryarXiv:2606. 06760v1 Announce Type: new Abstract: Medical large vision-language models (Med-LVLMs) have recently achieved remarkable progress in vision-language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for achieving clinically reasoning that connects visual findings with semantic interpretation.
We present MedSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MedSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into representation space of the language model. We further propose a medical region codebook into the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions.
These codes are decoded through the Region Perceiver to reconstruct segmentation mask, achieving end-to-end spatial grounding. Lastly, MedSIGHT combines Region Perceiver, Codebook and LLM using our proposed progressive training strategy to gradually aligns these modules stably. Trained on only 72K multimodal instruction pairs, MedSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.