Early-Token Confidence Predicts Reasoning Quality in Multi-Agent LLM Debate
Quick Answer
This paper shows that Early-token confidence, derived from token-level log-probabilities, is a strong predictor of reasoning quality in multi-agent LLM debates, outperforming full-sequence statistics.
Quick Take
Early-token confidence, derived from token-level log-probabilities, is a strong predictor of reasoning quality in multi-agent LLM debates, outperforming full-sequence statistics. This finding emphasizes the importance of the initial generation phase, particularly for supportive reasoning roles, in assessing reasoning reliability.
Key Points
- Early-token confidence consistently predicts reasoning quality in LLM debates.
- Log-probability trajectories reveal the initial generation phase is most informative.
- Supportive reasoning shows stronger alignment between confidence and quality than adversarial critique.
- The study utilizes a debate-based essay scoring framework for evaluation.
- Findings suggest lightweight methods for estimating reasoning reliability in LLM systems.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2606. 10307v1 Announce Type: new Abstract: Evaluating reasoning quality in multi-agent LLM systems is challenging, especially for open-ended tasks without reference answers. We investigate whether intrinsic confidence signals, token-level log-probabilities from decoding, can predict reasoning quality as assessed by LLM-as-judge evaluation. Using a debate-based essay scoring framework, we compare confidence proxies against rubric-based judge scores across two ASAP essay sets.
We find that early-token confidence, particularly within the first few generated tokens, is consistently the strongest predictor of reasoning quality, outperforming full-sequence statistics. Analysis of log-probability trajectories shows that the opening phase of generation is the most heterogeneous and therefore most informative. We also observe a systematic asymmetry between agent roles, with stronger alignment between confidence and quality for supportive reasoning than for adversarial critique.
These results suggest that early decoding dynamics provide a lightweight and effective signal for estimating reasoning reliability in multi-agent LLM systems.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The REFLECT benchmark reveals that current LLM judges are unreliable, achieving below 55% accuracy in evaluating reasoning and evidence use, highlighting the need for improved evaluation methods for deep research agents.