Spectral-LSH: Sub-Quadratic Prompt Compression via Krylov-Projected Locality-Sensitive Hashing
Quick Answer
Spectral-LSH introduces a training-free prompt compression method that reduces the quadratic scaling of long-prompt inference.
Quick Take
Evaluating models like Qwen2.5-7B and Qwen2.5-14B, it achieves significant compression ratios, improving performance metrics while maintaining quality. Notably, at a 16x compression, Qwen2.5-7B reduces PPL from 353.409 to 196.963.
Key Points
- Spectral-LSH uses Krylov subspace methods for efficient prompt compression.
- Compression ratios show a phase transition, with optimal performance at specific thresholds.
- At 8x compression, local LSH outperforms chunking across all metrics.
- Qwen2.5-14B achieves a PPL reduction from 9.533 to 3.427 at 16x compression.
- Adaptive backend balances chunking and spectral clustering for varying compression needs.
Paper Resources
Source Excerpt
Long-prompt inference remains expensive because prefill attention scales quadratically with sequence length. We propose Spectral-LSH, a training-free prompt compression method that operates before the prompt enters the language model. Spectral-LSH approximates the dominant components of an implicit attention-kernel operator using a Krylov subspace method together with random features, avoiding explicit $O(N^2)$ attention-kernel materialization. It then applies SimHash in the resulting attention
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.