LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation
Quick Answer
LEAP introduces a novel training curriculum for Vision Transformer distillation, enhancing student model performance by utilizing adaptive difficulty in feature learning.
Quick Take
The LEAP-distilled ViT-S achieves 90.1% accuracy on ImageNet-100, marking a 12.24% improvement over baselines, while reducing training FLOPs by 25.1% and time by 21%. This approach addresses the teacher-student gap in knowledge distillation, making it suitable for edge deployment.
Key Points
- LEAP utilizes intermediate feature maps for progressive knowledge distillation in ViTs.
- ViT-S achieves 90.1% accuracy on ImageNet-100, a 12.24% improvement.
- Adaptive difficulty selection accelerates convergence across various model sizes.
- Training FLOPs and time are reduced by 25.1% and 21%, respectively.
- Code is available at https://github.com/KevinZ0217/LEAP.
Paper Resources
Source Excerpt
Vision Foundation Models (VFMs) with Vision Transformer (ViT) backbones, such as DINOv2, have become essential for downstream tasks like object recognition and semantic segmentation. The immense computational requirements of backbones often necessitate distillation into smaller architectures for edge deployment. Feature-based knowledge distillation (KD) often suffers from the teacher-student gap; the student struggles to imitate teacher's complex feature map due to its limited capacity. To mitig
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →ProMoE-FL: Prototype-conditioned Mixture of Experts for Multimodal Federated Learning with Missing Modalities
ProMoE-FL introduces a Prototype-conditioned Mixture-of-Experts framework for multimodal federated learning, effectively addressing missing modalities. It outperforms existing methods on four chest X-ray datasets, demonstrating superior feature synthesis capabilities in both homogeneous and heterogeneous settings.