Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
Quick Answer
The paper introduces Stepwise Confidence Attribution (SCA), a framework for diagnosing multi-step reasoning failures in black-box LLMs without internal access.
Quick Take
SCA utilizes the Information Bottleneck principle to assess step-level confidence, leading to a 13.5% improvement in correction success rates over traditional answer-level feedback. Experiments demonstrate its effectiveness in identifying low-confidence steps correlated with reasoning errors.
Key Points
- SCA assigns confidence based on reasoning traces, improving error diagnosis in .
- Two methods: NIBS for non-parametric consistency and GIBS for graph-based learning.
- Identifies low-confidence steps strongly correlated with reasoning errors.
- Achieves up to 13.5% higher correction success rates using step-level feedback.
- Applicable to closed-source LLMs, enhancing diagnostic capabilities.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large Language Models have achieved strong performance on reasoning tasks with objective answers by generating step-by-step solutions, but diagnosing where a multi-step reasoning trace might fail remains difficult. Confidence estimation offers a diagnostic signal, yet existing methods are restricted to final answers or require internal model access. In this paper, we introduce Stepwise Confidence Attribution (SCA), a framework for closed-source LLMs that assigns step-level confidence based only on generated reasoning traces. SCA applies the Information Bottleneck principle: steps aligning with consensus structures across correct solutions receive high confidence, while deviations are flagged as potentially erroneous. We propose two complementary methods: (1) NIBS, a non-parametric IB approach measuring consistency without graph structures, and (2) GIBS, a graph-based IB model that learns subgraphs through a differentiable mask to capture logical variability. Extensive experiments on mathematical reasoning and multi-hop question answering show that SCA reliably identifies low-confidence steps strongly correlated with reasoning errors. Moreover, using step-level confidence to guide self-correction improves the correction success rate by up to 13.5\% over answer-level feedback.
| Comments: | Accepted by ICML 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI); Information Theory (cs.IT); Machine Learning (cs.LG) |
| MSC classes: | 68T50, 68T37, 68Q32 |
| ACM classes: | I.2.7; I.2.6; I.2.4 |
| Cite as: | arXiv:2605.19228 [cs.CL] |
| (or arXiv:2605.19228v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.19228 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Hua Wei [view email]
[v1]
Tue, 19 May 2026 00:57:51 UTC (520 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.