Energy-Efficient On-Device RAG on a Mobile NPU: System Design and Benchmark on Snapdragon X Elite
Quick Answer
This paper shows that The Snapdragon X Elite's Hexagon NPU enables an energy-efficient, end-to-end Retrieval-Augmented Generation (RAG) pipeline, achieving 9.1x higher embedding throughput and 12.3x less energy compared to CPU.
Quick Take
In benchmarks, it delivers 18.1x faster prefilling and 4.0x lower latency, maintaining answer quality on par with CPU and GPU. This innovation paves the way for sustainable edge intelligence across mobile NPUs.
Key Points
- NPU achieves 9.1x higher embedding throughput than CPU.
- Delivers 18.1x faster LLM prefilling on a 120-query benchmark.
- Uses 4.0x less system energy compared to CPU for end-to-end queries.
- Maintains answer quality comparable to CPU and GPU in evaluations.
- Sets a precedent for energy-efficient mobile AI solutions.
Paper Resources
Source Excerpt
arXiv:2606. 11257v1 Announce Type: new Abstract: (RAG) pipelines are compute-intensive, combining embedding, retrieval, reranking, and (LLM) generation. Running them entirely on-device benefits privacy, latency, and offline use, but the energy cost of CPU inference is a major barrier.
We present what is, to our knowledge, the first end-to-end RAG pipeline that runs all neural stages -- embedding, reranking, and LLM generation -- on the Qualcomm Hexagon NPU of the Snapdragon X Elite. Profiling on a Dell XPS 13 laptop, we compare NPU-accelerated RAG against CPU and OpenCL/Adreno GPU baselines on indexing and query workloads. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.
