LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation
Quick Answer
LEAP introduces a novel training curriculum for Vision Transformer distillation, enhancing student model performance by utilizing adaptive difficulty in feature learning.
Quick Take
The LEAP-distilled ViT-S achieves 90.1% accuracy on ImageNet-100, marking a 12.24% improvement over baselines, while reducing training FLOPs by 25.1% and time by 21%. This approach addresses the teacher-student gap in knowledge distillation, making it suitable for edge deployment.
Key Points
- LEAP utilizes intermediate feature maps for progressive knowledge distillation in ViTs.
- ViT-S achieves 90.1% accuracy on ImageNet-100, a 12.24% improvement.
- Adaptive difficulty selection accelerates convergence across various model sizes.
- Training FLOPs and time are reduced by 25.1% and 21%, respectively.
- Code is available at https://github.com/KevinZ0217/LEAP.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Vision Foundation Models (VFMs) with Vision Transformer (ViT) backbones, such as DINOv2, have become essential for downstream tasks like object recognition and semantic segmentation. The immense computational requirements of backbones often necessitate distillation into smaller architectures for edge deployment. Feature-based knowledge distillation (KD) often suffers from the teacher-student gap; the student struggles to imitate teacher's complex feature map due to its limited capacity. To mitigate this bottleneck, we propose LEAP: Layer-skipping Efficiency via Adaptive Progression, a training curriculum for ViT feature-based knowledge distillation. By utilizing the teacher's intermediate feature maps as a sequence of progressively more difficult targets, our curriculum allows the student to build a foundational representation before tackling higher-level abstractions. Our results demonstrate that this paradigm significantly accelerates convergence through adaptive difficulty selection across various student model sizes and dataset scales. With our curriculum, the LEAP-distilled ViT-S achieves 90.1% accuracy on ImageNet-100, a +12.24% improvement compared with baseline. On ImageNet-1K, LEAP achieves +3.84% and +7.75% improvement for the instance retrieval task on the Oxford and Paris datasets, respectively. Furthermore, the curriculum enables 25.1% savings in training FLOPs and 21% savings in training time on ImageNet-100 by implementing early-stopping for teacher inference during the initial stages of training. Code is available at this https URL
| Subjects: | Computer Vision and Pattern Recognition (cs.CV) |
| Cite as: | arXiv:2606.19483 [cs.CV] |
| (or arXiv:2606.19483v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.19483 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jiaqi Zhang [view email]
[v1]
Wed, 17 Jun 2026 18:14:51 UTC (521 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.