SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding
Quick Answer
SENSE introduces a novel approach to Retrieval-based Speculative Decoding (RSD) by leveraging semantic alignment through hidden states of target models, achieving up to 4.09 mean acceptance length and 3.26x speedup on LLaMA and Qwen families, while maintaining generation quality.
Quick Take
This method addresses the limitations of rigid lexical dependencies in existing RSD techniques.
Key Points
- SENSE enhances RSD by anchoring retrieval on target model's hidden states.
- Achieves 4.09 mean acceptance length and 3.26x speedup over baselines.
- Validates semantic equivalence through a Soft-gated Evaluation module.
- Extensive experiments show superior performance across diverse domains.
- Code will be released upon publication for further research.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 00021v1 Announce Type: new Abstract: Speculative Decoding (SD) accelerates (LLM) inference by employing a lightweight draft model to propose candidate tokens, which are verified in parallel by the target model, without compromising generation quality. While Retrieval-based Speculative Decoding (RSD) is favored for its plug-and-play versatility, its potential is impeded by rigid lexical dependencies, rendering both retrieval and verification brittle to surface-level variations.
To address this, we propose SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). By anchoring retrieval on the hidden states of the target model, SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence rather than surface forms. To ensure rigorous benchmarking, we deconstruct existing methods into atomic primitives within a unified framework, facilitating granular, component-level comparison.
Extensive experiments across diverse domains demonstrate that SENSE outperforms multiple baselines on the LLaMA and Qwen families, attaining up to 4. 09 mean acceptance length and 3. 26x speedup, while preserving generation quality. Our code will be released upon publication.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.