Benchmarking LLM Competence on Logical Inference over Probability Operators
Quick Answer
This paper shows that A new benchmark evaluates 29 large language models on logical inference over probability operators, revealing that most exhibit answer biases and only 9 surpass random chance.
Quick Take
The benchmark includes 14,320 prompts across 15 templates, highlighting systematic preferences for Yes or No answers, regardless of logical form.
Key Points
- Benchmark includes 14,320 procedurally-generated English prompts across 15 inference templates.
- 29 models evaluated, with only 9 exceeding random chance in accuracy.
- Most models show systematic biases favoring Yes or No answers.
- Variations in question form and names reveal biases across multiple axes.
- Implications are significant for high-stakes domains like medicine and law.
DeepSignal Analysis
What happened
A benchmark was introduced to evaluate 29 large language models on logical inference involving probability operators. The study found that most models exhibited answer biases, with only 9 models performing better than random chance across 14,320 prompts. The results indicate a systematic preference for Yes or No answers, regardless of the logical structure of the questions.
Key evidence
- The benchmark included 14,320 procedurally-generated English prompts across fifteen inference templates, systematically varying question form and content.
- Out of 29 evaluated models, only 9 exceeded random chance in their performance on logical inference tasks.
- The study identified systematic answer biases in most models, showing a preference for Yes or No answers independent of logical form.
Why it matters
The findings highlight significant limitations in the logical reasoning capabilities of current large language models, particularly in high-stakes fields like medicine and law where accurate inference is critical. The systematic biases observed may affect the reliability of these models in practical applications, raising concerns about their deployment in sensitive contexts. Understanding these limitations is essential for future improvements in model design and evaluation.
Paper Resources
Source Excerpt
Both expressions of uncertainty and inferences are ubiquitous in natural language, and valid inferences over natural-language expressions of uncertainty are necessary for not only everyday conversations but also for high-stakes domains such as medicine and law. While are increasingly evaluated on logical reasoning tasks, disentangling principled, symbolic reasoning from clever surface-level pattern matching is fraught with difficulty. We introduce a benchmark for reasoning
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.