CacheWeaver: Cache-Aware Evidence Ordering for Efficient Grounded RAG Inference
Quick Answer
CacheWeaver introduces a prompt-layer method that enhances retrieval-augmented generation (RAG) by optimizing evidence ordering, achieving a 20-33% reduction in median time-to-first-token (TTFT) across three vLLM configurations without compromising answer quality.
Quick Take
This approach leverages a prefix tree for efficient cache-aware ordering, significantly improving performance in grounded generation tasks.
Key Points
- CacheWeaver reduces median TTFT by 20-33% in grounded generation tasks.
- Utilizes a prefix tree to optimize evidence ordering without altering retrieval sets.
- Achieves 97.5% of the TTFT gain compared to oracle ordering with a simple scheduling layer.
- Improves efficiency in (RAG) applications.
- Maintains answer quality during performance enhancements across vLLM configurations.
Paper Resources
Source Excerpt
(RAG) improves factual grounding, but it also lengthens prompts and raises prefill cost. Prefix caching in serving engines such as vLLM reduces this cost only when requests share the same token prefix. In grounded generation, however, adjacent queries may retrieve overlapping evidence in different orders, so set overlap does not become reusable prefix overlap. We present CacheWeaver, a lightweight prompt-layer method for cache-aware evidence ordering. The method ke
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.