LEAP: Layer-skipping Efficiency via Adaptive Progression for Vision Transformer Distillation
Quick Answer
LEAP introduces a novel training curriculum for Vision Transformer distillation, enhancing student model performance by utilizing adaptive difficulty in feature learning.
Quick Take
LEAP introduces a novel training curriculum for Vision Transformer distillation, enhancing student model performance by utilizing adaptive difficulty in feature learning. The LEAP-distilled ViT-S achieves 90.1% accuracy on ImageNet-100, marking a 12.24% improvement over baselines, while reducing training FLOPs by 25.1% and time by 21%. This approach addresses the teacher-student gap in knowledge distillation, making it suitable for edge deployment.
Key Points
- LEAP utilizes intermediate feature maps for progressive knowledge distillation in ViTs.
- ViT-S achieves 90.1% accuracy on ImageNet-100, a 12.24% improvement.
- Adaptive difficulty selection accelerates convergence across various model sizes.
- Training FLOPs and time are reduced by 25.1% and 21%, respectively.
- Code is available at https://github.com/KevinZ0217/LEAP.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 19483v1 Announce Type: new Abstract: Vision Foundation Models (VFMs) with Vision Transformer (ViT) backbones, such as DINOv2, have become essential for downstream tasks like object recognition and semantic segmentation. The immense computational requirements of backbones often necessitate distillation into smaller architectures for edge deployment.
Feature-based knowledge distillation (KD) often suffers from the teacher-student gap; the student struggles to imitate teacher's complex feature map due to its limited capacity. To mitigate this bottleneck, we propose LEAP: Layer-skipping Efficiency via Adaptive Progression, a training curriculum for ViT feature-based knowledge distillation.
By utilizing the teacher's intermediate feature maps as a sequence of progressively more difficult targets, our curriculum allows the student to build a foundational representation before tackling higher-level abstractions. Our results demonstrate that this paradigm significantly accelerates convergence through adaptive difficulty selection across various student model sizes and dataset scales. With our curriculum, the LEAP-distilled ViT-S achieves 90. 1% accuracy on ImageNet-100, a +12.
24% improvement compared with baseline. On ImageNet-1K, LEAP achieves +3. 84% and +7. 75% improvement for the instance retrieval task on the Oxford and Paris datasets, respectively. Furthermore, the curriculum enables 25. 1% savings in training FLOPs and 21% savings in training time on ImageNet-100 by implementing early-stopping for teacher inference during the initial stages of training. Code is available at https://github. com/KevinZ0217/LEAP
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.