HeraSys: Collaborative Serving of Multiple LLM Workflows via Fine-Grained End-to-End Optimization
Quick Answer
This paper shows that HeraSys optimizes concurrent LLM workflows through fine-grained orchestration, achieving up to 2.17x reduction in P99 latency and 1.85x increase in throughput.
Quick Take
The system employs a load-aware scheduling policy and resource skewing to enhance performance while maintaining low average latency.
Key Points
- HeraSys reduces P99 latency by up to 2.17x under strict latency guarantees.
- The system increases serving throughput by up to 1.85x compared to existing solutions.
- It employs structural node merging and reuse to eliminate cross-workflow redundancy.
- A load-aware joint scheduling policy dynamically manages execution order.
- Extensive experiments validate HeraSys's performance improvements in multi-tenant environments.
DeepSignal Analysis
What happened
HeraSys is a new LLM serving system that focuses on optimizing the performance of concurrent workflows. It achieves significant improvements in latency and throughput through fine-grained orchestration and a load-aware scheduling policy. The system reportedly reduces P99 latency by up to 2.17 times and increases throughput by up to 1.85 times.
Key evidence
- HeraSys employs structural node merging and reuse to eliminate computational redundancy across workflows.
- The system introduces a load-aware joint scheduling policy that manages execution order based on inter- and intra-query priorities.
- Extensive experiments show that HeraSys can reduce P99 latency by up to 2.17 times while maintaining low average latency.
Why it matters
The ability to optimize inter-workflow performance is crucial as the demand for concurrent LLM workflows increases. HeraSys's approach could lead to more efficient resource utilization in multi-tenant environments, which is essential for scaling AI applications. The reported improvements in latency and throughput could significantly enhance user experience and operational efficiency in deploying LLMs.
What to watch
Paper Resources
📖 Reader Mode
~2 min readAbstract:The proliferation of Large Language Models (LLMs) has shifted serving systems from processing isolated requests to orchestrating high-concurrency, multi-tenant agentic workflows. However, existing solutions typically prioritize intra-workflow optimization, largely neglecting the significant potential for inter-workflow optimization. In this paper, we propose HeraSys, an LLM serving system designed to optimize the end-to-end performance of concurrent workflows. Through fine-grained orchestration, HeraSys eliminates cross-workflow computational redundancy via structural node merging and reuse. Furthermore, HeraSys introduces a load-aware joint scheduling policy that dynamically manages execution order by evaluating both inter- and intra-query priorities. By integrating a resource skewing mechanism with adaptive batching and pipeline decomposition, HeraSys effectively mitigates tail latency while maintaining low average latency, thereby substantially improving system throughput. Extensive experiments demonstrate that HeraSys reduces P99 latency by up to 2.17$\times$ and increases serving throughput by up to 1.85$\times$ under strict latency guarantees.
| Comments: | to be published in ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.22578 [cs.AI] |
| (or arXiv:2607.22578v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.22578 arXiv-issued DOI via DataCite |
Submission history
From: Zhiqing Tang [view email]
[v1]
Sun, 7 Jun 2026 03:14:57 UTC (1,046 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.