Benchmarking LLM Competence on Logical Inference over Probability Operators
Quick Answer
This paper shows that A new benchmark evaluates 29 large language models on logical inference over probability operators, revealing that most exhibit answer biases and only 9 surpass random chance.
Quick Take
The benchmark includes 14,320 prompts across 15 templates, highlighting systematic preferences for Yes or No answers, regardless of logical form.
Key Points
- Benchmark includes 14,320 procedurally-generated English prompts across 15 inference templates.
- 29 models evaluated, with only 9 exceeding random chance in accuracy.
- Most models show systematic biases favoring Yes or No answers.
- Variations in question form and names reveal biases across multiple axes.
- Implications are significant for high-stakes domains like medicine and law.
DeepSignal Analysis
What happened
A benchmark was introduced to evaluate 29 large language models on logical inference involving probability operators. The study found that most models exhibited answer biases, with only 9 models performing better than random chance across 14,320 prompts. The results indicate a systematic preference for Yes or No answers, regardless of the logical structure of the questions.
Key evidence
- The benchmark included 14,320 procedurally-generated English prompts across fifteen inference templates, systematically varying question form and content.
- Out of 29 evaluated models, only 9 exceeded random chance in their performance on logical inference tasks.
- The study identified systematic answer biases in most models, showing a preference for Yes or No answers independent of logical form.
Why it matters
The findings highlight significant limitations in the logical reasoning capabilities of current large language models, particularly in high-stakes fields like medicine and law where accurate inference is critical. The systematic biases observed may affect the reliability of these models in practical applications, raising concerns about their deployment in sensitive contexts. Understanding these limitations is essential for future improvements in model design and evaluation.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While large language models are increasingly evaluated on logical reasoning tasks, disentangling principled, symbolic reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a benchmark for reasoning over probability operators--inference over sentences with gradable epistemic modals (e.g., probably, might, must) containing 14,320 procedurally-generated English prompts across fifteen inference templates, systematically varying question form, negation strategy, and surface content. Evaluating 29 models, we find that most show answer biases independent of the logical form, a systematic preference for Yes or No. We summarize this with a competence floor: the worse of a model's accuracy on Yes-correct and No-correct items. Only 9 of 29 models exceed random chance. We also test variations in question form, verb phrases/activity, and both the gender and origin of names used in the prompts, finding biases across every axis.
| Comments: | Under review |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.27405 [cs.CL] |
| (or arXiv:2607.27405v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.27405 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Alvin Grissom II [view email]
[v1]
Wed, 29 Jul 2026 19:22:09 UTC (231 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.