Sticky Routing: Training MoE Models for Memory-Efficient Inference
Quick Answer
StickyMoE introduces a differentiable routing consistency loss to reduce expert switching in Mixture-of-Experts models during training.
Quick Take
StickyMoE introduces a differentiable routing consistency loss to reduce expert switching in Mixture-of-Experts models during training. This method decreases the expert switch rate by up to 60% with minimal perplexity degradation (<4%), outperforming traditional post-hoc fine-tuning approaches. The technique enhances memory efficiency for edge devices without requiring architectural changes.
Key Points
- StickyMoE penalizes abrupt expert switches between adjacent tokens.
- Requires no architectural changes, adding only a single hyperparameter.
- Reduces expert switch rate by up to 60% during training.
- Achieves less than 4% perplexity degradation compared to existing methods.
- Enhances routing temporal locality efficiently at training time.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Mixture-of-Experts (MoE) models activate only a sparse subset of experts per token, yet consecutive tokens frequently activate different experts -- causing constant weight swapping between slow storage and fast memory on edge devices. Existing remedies are either system-level (caching heuristics) or post-hoc (router fine-tuning), leaving the root cause unchanged during pretraining. We propose StickyMoE, a differentiable routing consistency loss that penalises abrupt expert switches between adjacent tokens, encouraging the router to maintain the same expert assignment across semantically coherent spans. StickyMoE requires no architectural changes, adds a single hyperparameter lambda, and unlike post-hoc methods, allows expert representations and routing decisions to co-adapt from the first training step. Experiments on small-scale MoE language models show that StickyMoE reduces the expert switch rate by up to 60% with less than 4% perplexity degradation, Pareto-dominating post-hoc fine-tuning on the quality-locality frontier. Routing temporal locality is most efficiently instilled at training time.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2607.08780 [cs.LG] |
| (or arXiv:2607.08780v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2607.08780 arXiv-issued DOI via DataCite |
Submission history
From: Ali Kayyam [view email]
[v1]
Fri, 12 Jun 2026 23:47:19 UTC (513 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.