HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
Quick Answer
This paper shows that HeraSys optimizes concurrent LLM workflows through fine-grained orchestration, achieving up to 2.17x reduction in P99 latency and 1.85x increase in throughput.
Quick Take
The system employs a load-aware scheduling policy and resource skewing to enhance performance while maintaining low average latency.
Key Points
- HeraSys reduces P99 latency by up to 2.17x under strict latency guarantees.
- The system increases serving throughput by up to 1.85x compared to existing solutions.
- It employs structural node merging and reuse to eliminate cross-workflow redundancy.
- A load-aware joint scheduling policy dynamically manages execution order.
- Extensive experiments validate HeraSys's performance improvements in multi-tenant environments.
DeepSignal Analysis
What happened
HeraSys is a new LLM serving system that focuses on optimizing the performance of concurrent workflows. It achieves significant improvements in latency and throughput through fine-grained orchestration and a load-aware scheduling policy. The system reportedly reduces P99 latency by up to 2.17 times and increases throughput by up to 1.85 times.
Key evidence
- HeraSys employs structural node merging and reuse to eliminate computational redundancy across workflows.
- The system introduces a load-aware joint scheduling policy that manages execution order based on inter- and intra-query priorities.
- Extensive experiments show that HeraSys can reduce P99 latency by up to 2.17 times while maintaining low average latency.
Why it matters
The ability to optimize inter-workflow performance is crucial as the demand for concurrent LLM workflows increases. HeraSys's approach could lead to more efficient resource utilization in multi-tenant environments, which is essential for scaling AI applications. The reported improvements in latency and throughput could significantly enhance user experience and operational efficiency in deploying LLMs.
What to watch
Paper Resources
Source Excerpt
The proliferation of (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows. However, existing solutions typically prioritize intra-workflow optimization, largely neglecting the significant potential for inter-workflow optimization. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end-to-end performance of concurrent workflows. Through fine-grained orchestration,
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.