Sticky Routing: Training MoE Models for Memory-Efficient Inference
Quick Answer
StickyMoE introduces a differentiable routing consistency loss to reduce expert switching in Mixture-of-Experts models during training.
Quick Take
This method decreases the expert switch rate by up to 60% with minimal perplexity degradation (<4%), outperforming traditional post-hoc fine-tuning approaches. The technique enhances memory efficiency for edge devices without requiring architectural changes.
Key Points
- StickyMoE penalizes abrupt expert switches between adjacent tokens.
- Requires no architectural changes, adding only a single hyperparameter.
- Reduces expert switch rate by up to 60% during training.
- Achieves less than 4% perplexity degradation compared to existing methods.
- Enhances routing temporal locality efficiently at training time.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.08780 [cs.LG] |
| (or arXiv:2607.08780v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08780 arXiv-issued DOI via DataCite |
Submission history
From: Ali Kayyam [view email]
[v1]
Fri, 12 Jun 2026 23:47:19 UTC (513 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.