SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding
Quick Answer
SENSE introduces a novel approach to Retrieval-based Speculative Decoding (RSD) by leveraging semantic alignment through hidden states of target models, achieving up to 4.09 mean acceptance length and 3.26x speedup on LLaMA and Qwen families, while maintaining generation quality.
Quick Take
SENSE introduces a novel approach to Retrieval-based Speculative Decoding (RSD) by leveraging semantic alignment through hidden states of target models, achieving up to 4.09 mean acceptance length and 3.26x speedup on LLaMA and Qwen families, while maintaining generation quality. This method addresses the limitations of rigid lexical dependencies in existing RSD techniques.
Key Points
- SENSE enhances RSD by anchoring retrieval on target model's hidden states.
- Achieves 4.09 mean acceptance length and 3.26x speedup over baselines.
- Validates semantic equivalence through a Soft-gated Evaluation module.
- Extensive experiments show superior performance across diverse domains.
- Code will be released upon publication for further research.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 00021v1 Announce Type: new Abstract: Speculative Decoding (SD) accelerates Large Language Model (LLM) inference by employing a lightweight draft model to propose candidate tokens, which are verified in parallel by the target model, without compromising generation quality. While Retrieval-based Speculative Decoding (RSD) is favored for its plug-and-play versatility, its potential is impeded by rigid lexical dependencies, rendering both retrieval and verification brittle to surface-level variations.
To address this, we propose SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). By anchoring retrieval on the hidden states of the target model, SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence rather than surface forms. To ensure rigorous benchmarking, we deconstruct existing methods into atomic primitives within a unified framework, facilitating granular, component-level comparison.
Extensive experiments across diverse domains demonstrate that SENSE outperforms multiple baselines on the LLaMA and Qwen families, attaining up to 4. 09 mean acceptance length and 3. 26x speedup, while preserving generation quality. Our code will be released upon publication.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.