SpecLA: Efficient Speculative Decoding for Linear-Attention Models
Quick Answer
SpecLA introduces an efficient speculative decoding runtime for stateful linear-attention models, achieving up to 1.70x speedup over traditional autoregressive decoding on NVIDIA H100 with GDN-1.3B target.
Quick Take
It employs topology-aware kernels and confidence pruning to optimize the verification of token candidates, enhancing performance in language tasks.
Key Points
- SpecLA optimizes speculative decoding for linear-attention models, enhancing efficiency.
- Achieves up to 1.70x speedup on NVIDIA H100 with GDN-1.3B benchmark.
- Utilizes topology-aware kernels for verifying token chains and trees.
- Implements confidence pruning to filter useful candidates for verification.
- Addresses limitations of existing speculative systems designed for Transformer KV caches.
Paper Resources
Source Excerpt
Linear-attention models replace the growing KV cache with recurrent states, but autoregressive decoding still reads, updates, and writes these states one token at a time. Speculative decoding can reduce this cost by verifying several draft tokens in one target pass, yet existing speculative systems are designed for Transformer KV caches. For stateful linear-attention targets, verification must follow recurrent dependencies across chains and branches, acceptance must update only the accepted stat
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.
