Quantifying Consistency in LLM Logical Reasoning via Structural Uncertainty
Quick Answer
This study introduces structural uncertainty as a framework to evaluate logical reasoning consistency in large language models (LLMs).
Quick Take
By analyzing pairwise preferences among candidate solutions across five and eight benchmarks, it reveals that within-trial ambiguity correlates positively with correctness, while across-trial instability indicates unreliable reasoning, enhancing the identification of unreliable instances in logical and mathematical tasks.
Key Points
- Structural uncertainty assesses consistency in LLM reasoning beyond output dispersion.
- The framework uses pairwise preferences to rank candidate solutions effectively.
- Within-trial ambiguity correlates positively with correctness in reasoning tasks.
- Across-trial instability signals unreliable reasoning paths in LLM outputs.
- The approach improves identification of unreliable instances across various benchmarks.
Paper Resources
Source Excerpt
arXiv:2606. 17312v1 Announce Type: new Abstract: can arrive at the same answer through reasoning paths that are unstable, contradictory, or difficult to rank consistently -- a failure mode especially prevalent in multi-step deductive reasoning. Existing methods assess reliability primarily through output dispersion -- measuring how much sampled answers differ -- but this discards a complementary signal: whether the model can consistently rank competing reasoning candidates.
We propose structural uncertainty, a consistency-aware framework derived from the stability of self-preference-induced rankings over sampled reasoning solutions. …
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.