Beyond Accuracy and Cost: Latency-Aware LLM Query Routing for Dynamic Workloads
Quick Answer
This paper presents a latency-aware query routing system for language models, improving accuracy-cost utility by up to 40% while maintaining standard latencies.
Quick Take
The proposed lightweight latency estimator simulates token processing to optimize query assignments based on latency, accuracy, and cost, addressing the limitations of current latency-agnostic routers.
Key Points
- Current query routers ignore latency, relying on basic load-balancing methods.
- The new system estimates time-to-first-token (TTFT) for improved routing decisions.
- Joint optimization of latency, accuracy, and cost leads to significant performance gains.
- Experimental results show a 40% improvement in accuracy-cost utility.
- The approach maintains latencies comparable to traditional methods.
DeepSignal Analysis
What happened
A new latency-aware query routing system for language models has been developed, which optimizes query assignments based on latency, accuracy, and cost. This system reportedly improves accuracy-cost utility by up to 40% while maintaining standard latencies. The approach addresses the limitations of existing latency-agnostic routers.
Key evidence
- Current query routers often ignore latency, relying on load-balancing methods like round-robin, which do not consider model accuracy or inference costs.
- The proposed latency estimator simulates autoregressive token batch processing to estimate time-to-first-token (TTFT) for queries.
- Experimental results show that the new routing system achieves a 40% improvement in accuracy-cost utility without increasing latencies compared to standard methods.
Why it matters
Incorporating latency into query routing can significantly enhance the performance of language models, particularly in dynamic workloads where response time is critical. This advancement could lead to more efficient resource utilization and improved user experiences in applications relying on language models. As demand for real-time processing grows, such innovations may become essential for maintaining competitive advantages in AI-driven services.
Paper Resources
Source Excerpt
Modern language query routers improve inference efficiency by assigning each query to a model that balances response quality and monetary cost. However, current query routers are largely latency-agnostic and do not consider the generation latency experienced by queries at model instances. In practice, latency is often controlled by load-balancing policies such as round-robin or join-the-shortest-queue, which do not account for model accuracy or inference cost. Incorporating query latency into ro
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →RAIL Guard: Closing the Evaluation-to-Remediation Gap in Responsible AI for Agents
RAIL Guard introduces a closed-loop AI pipeline for large language models (LLMs) that evaluates outputs across eight dimensions and iteratively remediates failures, achieving 96.9% convergence compared to 49.1% for traditional block-and-retry methods. The system reduces unsafe agent executions by 33% without impacting task completion and is available as open-source SDKs.