Routing Without Training: Controllable-Ratio LLM Offloading via Reliability Gating
Quick Answer
CARGO introduces a training-free routing framework for local-cloud collaboration in LLMs, leveraging inference-time agreement for decision-making.
Quick Take
It consistently outperforms training-based routers across various tasks and models, suggesting effective collaboration without additional training. This approach enhances resource efficiency and adaptability in deploying .
Key Points
- CARGO uses prompt-varied sampling to estimate inference-time agreement.
- Employs Bayesian early stopping for efficient uncertainty control.
- Supports arbitrary collaboration ratios with lightweight calibration.
- Outperforms training-free baselines and some supervised routers.
- Demonstrates effectiveness across diverse reasoning and question-answering tasks.
DeepSignal Analysis
What happened
CARGO is a new routing framework that enables local-cloud collaboration for large language models without requiring training. It utilizes inference-time agreement from local models to determine when to execute tasks locally or offload them to cloud models. This approach has shown to outperform traditional training-based routers across various tasks and models.
Key evidence
- CARGO estimates inference-time agreement through prompt-varied sampling, allowing it to make effective routing decisions without prior training.
- The framework consistently outperforms other training-free baselines and, in some cases, even surpasses supervised learned routers across diverse reasoning and question-answering tasks.
- CARGO supports arbitrary target collaboration ratios and employs Bayesian early stopping for efficient uncertainty control, enhancing resource efficiency in deploying large language models.
Why it matters
The introduction of CARGO could significantly reduce the resource demands typically associated with deploying large language models by eliminating the need for additional training. This could lead to more adaptable and efficient systems, particularly in environments with limited resources. The findings suggest that local models can autonomously determine their reliability, which may streamline operations in various applications.
Paper Resources
Source Excerpt
Local-cloud collaboration is a practical way to deploy under resource constraints, but existing methods often rely on trained routers or collaboration-aware finetuning that tie routing behavior to a particular operating regime. In this work, we show that such training may be unnecessary: the local model's own inference-time agreement across sampled responses already provides a strong signal for deciding when to trust local execution and when to offload to a stronger cloud m
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.