Multi-Teacher Contrastive Distillation for Edge-Efficient Pathology Foundation Models
Quick Answer
This paper shows that The MuCoDi framework distills embeddings from multiple pathology foundation models into compact encoders, achieving up to 71.0% AUROC with reduced model sizes.
Quick Take
The MuCoDi framework distills embeddings from multiple pathology foundation models into compact encoders, achieving up to 71.0% AUROC with reduced model sizes. MobileOne students on Raspberry Pi 5 demonstrate a 605-fold speedup over Virchow2 while maintaining competitive performance, enabling practical edge deployment in pathology.
Key Points
- MuCoDi distills frozen tile embeddings into lightweight MobileOne and RepViT models.
- RepViT-based MuCoEdge achieves 71.0% AUROC, close to Virchow2's 71.8%.
- MuCoEdge-R1.0 has only 6.4M parameters and 1.12 GFLOPs with 70.9% AUROC.
- MobileOne students achieve 605-fold speedup on Raspberry Pi 5 with 66.5% AUROC.
- Code for MuCoDi is publicly available for further research.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Computational pathology foundation models (PFMs) have advanced whole-slide image analysis. However, their size and inference cost hinder local deployment in pathology departments. We propose MuCoDi, a pretraining framework that distills frozen tile embeddings from multiple PFMs into compact edge-oriented encoders. Instead of regressing individual teacher features, MuCoDi trains lightweight MobileOne and RepViT students with a contrastive distillation objective adapted from MoCo v3, where cached Virchow2, UNI2, and H-Optimus-1 embeddings replace momentum-encoder keys. We pretrain students on 14.3M TCGA tiles from only 11.8K WSIs and evaluate frozen encoders on 23 clinically curated downstream classification tasks. RepViT-based MuCoEdge students retain near-teacher performance while reducing model size by orders of magnitude: MuCoEdge-R2.3 and MuCoEdge-R1.5 reach 71.0% external AUROC, within 0.8 percentage points of the best teacher (Virchow2, 71.8%), while MuCoEdge-R2.3 obtains the best external F1 and the second-best AUPRC (51.8% and 53.3%). MuCoEdge-R1.0 reaches 70.9% AUROC with only 6.4M parameters and 1.12 GFLOPs. On a Raspberry Pi 5, sub-million-parameter MobileOne students achieve up to 605-fold single-tile speedup over Virchow2 while retaining 66.5% to 66.9% external AUROC, demonstrating that PFM-quality pathology representations can be moved toward practical edge deployment. Code is available at this https URL.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2607.05533 [cs.CV] |
| (or arXiv:2607.05533v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05533 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Tim Lenz [view email]
[v1]
Mon, 6 Jul 2026 18:14:37 UTC (914 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.