SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding
Quick Take
SENSE introduces a novel approach to Retrieval-based Speculative Decoding (RSD) by leveraging semantic alignment through hidden states of target models, achieving up to 4.09 mean acceptance length and 3.26x speedup on LLaMA and Qwen families, while maintaining generation quality. This method addresses the limitations of rigid lexical dependencies in existing RSD techniques.
Key Points
- SENSE enhances RSD by anchoring retrieval on target model's hidden states.
- Achieves 4.09 mean acceptance length and 3.26x speedup over baselines.
- Validates semantic equivalence through a Soft-gated Evaluation module.
- Extensive experiments show superior performance across diverse domains.
- Code will be released upon publication for further research.
Article Content
From source RSS / original summaryarXiv:2606. 00021v1 Announce Type: new Abstract: Speculative Decoding (SD) accelerates Large Language Model (LLM) inference by employing a lightweight draft model to propose candidate tokens, which are verified in parallel by the target model, without compromising generation quality. While Retrieval-based Speculative Decoding (RSD) is favored for its plug-and-play versatility, its potential is impeded by rigid lexical dependencies, rendering both retrieval and verification brittle to surface-level variations.
To address this, we propose SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). By anchoring retrieval on the hidden states of the target model, SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence rather than surface forms. To ensure rigorous benchmarking, we deconstruct existing methods into atomic primitives within a unified framework, facilitating granular, component-level comparison.
Extensive experiments across diverse domains demonstrate that SENSE outperforms multiple baselines on the LLaMA and Qwen families, attaining up to 4. 09 mean acceptance length and 3. 26x speedup, while preserving generation quality. Our code will be released upon publication.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.