A World Model of Radiologist Reading for Medical Image Representation Learning
Quick Take
GazeWorld is a novel medical imaging model that predicts latent representations based on radiologist eye-tracking data, achieving state-of-the-art diagnostic accuracy on CheXpert, RSNA Pneumonia, and SIIM-ACR Pneumothorax benchmarks. It outperformed LogitGaze-Med by over 16% in ScanMatch and 22% in SED on the GazeSearch benchmark, demonstrating the potential of modeling expert reading behaviors for AI pretraining.
Key Points
- GazeWorld predicts the next fixated patch's representation autoregressively from prior patches.
- Achieved state-of-the-art accuracy across nine supervised settings on key medical benchmarks.
- Outperformed LogitGaze-Med by 16% in ScanMatch and 22% in SED on GazeSearch.
- Generates patch representations without requiring actual gaze data during inference.
- Demonstrates the value of modeling expert reading processes for medical imaging AI.
Article Content
From source RSS / original summaryarXiv:2605. 23992v1 Announce Type: new Abstract: Radiologist eye-tracking data provide a rich record of how experts search, compare, and accumulate evidence during image reading; yet, existing methods exploit this signal only partially, either as a static spatial prior or as an auxiliary prediction target decoupled from diagnosis. We propose GazeWorld, a medical imaging world model that treats the image as the world and the radiologist's fixation sequence as a trajectory through it.
GazeWorld autoregressively predicts the latent representation of the next fixated patch from all previously visited ones, while a spatial-completion branch covers unvisited regions. At inference, GazeWorld generates a sequence of patch representations from the image alone without requiring real gaze data.
Frozen GazeWorld features achieve state-of-the-art diagnostic accuracy across all nine supervised settings on CheXpert, RSNA Pneumonia, and SIIM-ACR Pneumothorax, as well as the highest zero-shot accuracy on all three benchmarks. On the GazeSearch benchmark, a generic decoder trained on the same frozen features outperforms the purpose-built LogitGaze-Med by over 16\% in ScanMatch and 22\% in SED, despite not being explicitly trained to predict gaze.
GazeWorld demonstrates that modeling how experts read, not just what they conclude, offers a promising pretraining paradigm for medical imaging AI.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →Evi-Steer: Learning to Steer Biomedical Vision-Language Models through Efficient and Generalizable Evidential Tuning
Evi-Steer introduces a novel evidential tuning framework for BiomedCLIP, achieving 0.11% parameter updates while enhancing uncertainty-aware fine-tuning. It outperforms state-of-the-art methods across 15 biomedical imaging datasets, proving effective in few-shot learning and domain shifts for clinical applications.
