MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies
Quick Answer
MicroSpec introduces a training-free method for speculative decoding that reduces vocabulary size by over 40x, achieving a 51.6% latency reduction compared to EAGLE-2.
Quick Take
MicroSpec introduces a training-free method for speculative decoding that reduces vocabulary size by over 40x, achieving a 51.6% latency reduction compared to EAGLE-2. This context-sensitive approach enhances performance on contemporary hardware without additional trained parameters, providing a 1.12-1.32x speedup across benchmarks.
Key Points
- MicroSpec reduces active vocabulary size to under 3k tokens from over 100k.
- Achieves 51.6% average reduction in draft inference latency.
- Provides 1.12-1.32x speedup over EAGLE-2 on various benchmarks.
- No additional trained parameters required for performance gains.
- Utilizes asynchronous gathering and GPU-resident state management.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2605. 26444v1 Announce Type: new Abstract: Large language models typically employ vocabularies of over 100k tokens, which creates a major computational bottleneck at the final linear projection layer when performing speculative decoding. Current methods for vocabulary pruning depend on either fixed or coarse-grained sub-vocabularies, requiring around 30k active tokens to preserve the quality of the draft model.
We introduce MicroSpec, a training-free technique that overcomes this limitation by building a compact, context-sensitive active vocabulary on the fly for every decoding step. Exploiting the natural temporal locality found in language generation, MicroSpec attains high token coverage while reducing the average vocabulary size by more than 40x (down to under 3k tokens), all without any additional trained parameters.
To translate this high sparsity into actual speedups on contemporary hardware, we present a co-designed system and algorithm that mitigates the overhead of sparse memory accesses via asynchronous gathering and GPU-resident state management. Acting as a plug-and-play enhancement, MicroSpec reduces draft inference latency by 51. 6% on average, achieving an end-to-end speedup of 1. 12-1.
32x relative to the leading speculative decoding approach EAGLE-2 on various benchmarks, while also surpassing more sophisticated training-based pruning baselines.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.