Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
Quick Answer
The proposed C3RL algorithm enhances the calibration of large language models (LLMs) by integrating correctness and confidence rewards, outperforming existing methods.
Quick Take
Coupled with the CAS strategy, it allows for adaptive resource allocation based on confidence, achieving up to 12.33 times reduction in inference costs while improving performance on 8 datasets.
Key Points
- C3RL integrates correctness and confidence rewards for better calibration.
- Outperforms state-of-the-art methods in performance and calibration metrics.
- CAS allocates computational resources based on response confidence.
- Achieves up to 12.33 times reduction in inference budget.
- Code, data, and models will be publicly released.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Training large language models (LLMs) with reinforcement learning (RL) has significantly advanced their performance on reasoning and question-answering tasks. However, prevailing RL reward designs typically prioritize response correctness, neglecting to incentivize models to express their confidence accurately. This leads to a critical problem: performance gains are often accompanied by poor calibration between confidence and accuracy, misleading models to overconfidently hallucinate when uncertain. To address this limitation, we propose $\textbf{C}$orrectness and $\textbf{C}$onfidence $\textbf{C}$alibration $\textbf{R}$einforcement $\textbf{L}$earning ($\textbf{C3RL}$), a novel RL algorithm integrating correctness, calibration and dataset-informed reference accuracy rewards together. Comprehensive evaluation across 8 text and multimodal datasets demonstrates that C3RL enhances calibration without sacrificing accuracy, outperforming the current state-of-the-art method in both performance and calibration metrics. Utilizing the well-calibrated verbalized confidence from C3RL, we further introduce $\textbf{C}$onfidence-based $\textbf{A}$daptive Test Time $\textbf{S}$caling ($\textbf{CAS}$), an adjustable inference-time strategy that allocates computational resources based on response confidence. Experiments show that CAS surpasses majority voting on both in-domain and out-of-domain datasets while reducing the inference budget by up to 12.33 times. We believe the synergy of C3RL and CAS paves the way for deploying more reliable and resource-efficient LLMs. The code, data and models will be released.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.01612 [cs.AI] |
| (or arXiv:2607.01612v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01612 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Xuqing Yang [view email]
[v1]
Thu, 2 Jul 2026 02:29:33 UTC (1,253 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.