MicroSpec: Accelerating Speculative Decoding with Lightweight In-Context Vocabularies
Quick Take
MicroSpec introduces a training-free method for speculative decoding that reduces vocabulary size by over 40x, achieving a 51.6% latency reduction compared to EAGLE-2. This context-sensitive approach enhances performance on contemporary hardware without additional trained parameters, providing a 1.12-1.32x speedup across benchmarks.
Key Points
- MicroSpec reduces active vocabulary size to under 3k tokens from over 100k.
- Achieves 51.6% average reduction in draft inference latency.
- Provides 1.12-1.32x speedup over EAGLE-2 on various benchmarks.
- No additional trained parameters required for performance gains.
- Utilizes asynchronous gathering and GPU-resident state management.
Article Content
From source RSS / original summaryarXiv:2605. 26444v1 Announce Type: new Abstract: Large language models typically employ vocabularies of over 100k tokens, which creates a major computational bottleneck at the final linear projection layer when performing speculative decoding. Current methods for vocabulary pruning depend on either fixed or coarse-grained sub-vocabularies, requiring around 30k active tokens to preserve the quality of the draft model.
We introduce MicroSpec, a training-free technique that overcomes this limitation by building a compact, context-sensitive active vocabulary on the fly for every decoding step. Exploiting the natural temporal locality found in language generation, MicroSpec attains high token coverage while reducing the average vocabulary size by more than 40x (down to under 3k tokens), all without any additional trained parameters.
To translate this high sparsity into actual speedups on contemporary hardware, we present a co-designed system and algorithm that mitigates the overhead of sparse memory accesses via asynchronous gathering and GPU-resident state management. Acting as a plug-and-play enhancement, MicroSpec reduces draft inference latency by 51. 6% on average, achieving an end-to-end speedup of 1. 12-1.
32x relative to the leading speculative decoding approach EAGLE-2 on various benchmarks, while also surpassing more sophisticated training-based pruning baselines.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.