CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection
Quick Answer
CLEAR-MoE introduces a four-phase pipeline to convert frozen Vision Transformers into sparse Mixture-of-Experts models, achieving 99.9% accuracy retention on Imagenette with DeiT-Small.
Quick Take
CLEAR-MoE introduces a four-phase pipeline to convert frozen Vision Transformers into sparse Mixture-of-Experts models, achieving 99.9% accuracy retention on Imagenette with DeiT-Small. The method utilizes shared low-rank SVD bases and lightweight routers, demonstrating minimal performance variation across different configurations. However, it incurs a 1.3-1.7x speed overhead compared to dense implementations due to routing complexities.
Key Points
- Achieves 99.9% accuracy retention on Imagenette with DeiT-Small.
- Utilizes shared low-rank SVD basis for expert extraction.
- Performance variation across configurations is minimal (<= 0.10%).
- Routing overhead makes CLEAR-MoE 1.3-1.7x slower than dense models.
- Identifies fused dispatch kernels as a future optimization direction.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We present CLEAR-MoE, a four-phase post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights. The pipeline (i) scores feed-forward network (FFN) layers by sparsity, clusterability, and output sensitivity; (ii) decomposes selected layers into a shared low-rank SVD basis and per-cluster residual experts using k-means clustering; (iii) trains lightweight routers supervised by cluster labels; and (iv) dispatches tokens through pluggable CUDA backends. On Imagenette with DeiT-Small, CLEAR-MoE retains 99.9% of the dense model's accuracy (86.70 +/- 0.02% versus 86.73%). Extensive ablation studies reveal a consistent empirical finding: the shared SVD basis is the primary factor responsible for preserving accuracy. Random routing, learned routing, and three different router architectures produce nearly identical performance, with accuracy varying by at most 0.06 percentage points (86.62%-86.68%). Accuracy also remains stable across different SVD ranks, expert counts (2-8), calibration set sizes (50-500), and random seeds. This behavior generalizes across five ViT backbones (DeiT-Tiny, DeiT-Small, DeiT-Base, ViT-Small, and ViT-Base), covering models from 5.7M to 86.6M parameters, with accuracy differences <= 0.10 percentage points from their dense counterparts. On a GTX 960 GPU, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense implementation. A dispatch microbenchmark further shows that routing is an order of magnitude more memory-bound than expert matrix multiplications, identifying fused dispatch kernels as a promising direction for future optimization.
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Distributed, Parallel, and Cluster Computing (cs.DC) |
| Cite as: | arXiv:2606.28516 [cs.CV] |
| (or arXiv:2606.28516v1 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2606.28516 arXiv-issued DOI via DataCite |
Submission history
From: Md. Irtiza Hossain [view email]
[v1]
Fri, 26 Jun 2026 18:12:03 UTC (1,494 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CV
See more →LLM-Guided ANN Index Optimization for Human-Object Interaction Retrieval
A phase-aware LLM agent optimizes human-object interaction retrieval, outperforming Optuna TPE by 33.3% and VDTuner by 34.2% on the HICO-DET benchmark. This method enhances throughput by 15.3x over UniIR and demonstrates strong transferability across vector database management systems.