UniScale: Adaptive Unified Inference Scaling via Online Joint Optimization of Model Routing and Test-Time Scaling
Quick Answer
UniScale introduces Unified Inference Scaling (UIS) to optimize model routing and test-time scaling for large language models, enhancing adaptability in dynamic environments.
Quick Take
By framing this as a contextual multi-armed bandit problem, it achieves a superior quality-cost trade-off, outperforming existing methods in diverse inference scenarios.
Key Points
- UniScale combines model routing and in a unified optimization framework.
- The framework uses LinUCB for learning inference policies effectively.
- It addresses limitations of existing methods by enabling fine-grained performance adjustments.
- Evaluation shows improved quality-cost trade-offs across various dynamic inference scenarios.
- Efficiency-aware learning ensures stable optimization over high-dimensional action spaces.
Paper Resources
📖 Reader Mode
~2 min readAbstract:In real-world deployments of large language models (LLMs), balancing inference quality and computational cost has become a central challenge. Existing approaches tackle this trade-off along two largely independent dimensions: model routing, which switches among models of different scales to match request complexity, and test-time scaling (TTS), which adjusts inference-time compute within a fixed model for fine-grained control. However, this decoupled design introduces inherent limitations. Model routing yields coarse-grained, discrete performance changes due to the sparse set of model scales, while single-model TTS often encounters capacity ceilings and exhibits diminishing returns as compute increases. Moreover, treating the two mechanisms separately restricts adaptability in dynamic inference environments. To overcome these limitations, we introduce Unified Inference Scaling (UIS), which unifies model routing and TTS in a single optimization space. Building on this formulation, we propose UniScale, an online framework that models adaptive UIS as a contextual multi-armed bandit problem and learns inference policies via LinUCB. The framework incorporates efficiency-aware learning and cost modeling to ensure stable and scalable optimization over high-dimensional action spaces. Evaluation shows that UniScale effectively exploits the synergy in the UIS space to deliver a fine-grained and consistently better quality-cost trade-off across diverse, dynamic inference scenarios.
| Comments: | Accepted at the 43rd International Conference on Machine Learning (ICML 2026) |
| Subjects: | Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.30898 [cs.AI] |
| (or arXiv:2605.30898v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2605.30898 arXiv-issued DOI via DataCite |
Submission history
From: Kaiyu Huang [view email]
[v1]
Fri, 29 May 2026 06:31:21 UTC (373 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.