IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs
Quick Answer
This paper shows that ISOSCI benchmark reveals that 91.3% of reasoning-mode gains in LLMs are knowledge-dependent, challenging the assumption that chain-of-thought reasoning enhances scientific problem-solving.
Quick Take
Notably, the reasoning-specialized model o3-mini outperformed on but underperformed on ISOSCI, indicating benchmark choice significantly influences conclusions about reasoning utility.
Key Points
- ISOSCI benchmarks separate reasoning ability from domain knowledge retrieval in evaluation.
- 91.3% of reasoning gains are knowledge-dependent, not structure-invariant.
- Reasoning toggles provide less than 5 percentage points accuracy gain across all domains.
- The o3-mini model outperforms on GPQA Diamond but underperforms on ISOSCI.
- Benchmark choice critically influences conclusions about LLM reasoning utility.
Paper Resources
📖 Reader Mode
~2 min readAbstract:We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation. Each pair shares identical logical structure but requires different domain-specific knowledge, enabling controlled attribution of reasoning-mode gains. Across five model pairs spanning four model families, we find that 91.3% of reasoning-mode gains are knowledge-dependent rather than structure-invariant (63/69 gains; Wilson 95% CI [82.3%, 96.0%]), directly challenging the assumption that chain-of-thought reasoning improves short-horizon procedural scientific problem-solving. Reasoning toggles on highly capable models provide less than 5 percentage points accuracy gain across all domains, and a reasoning-specialized model (o3-mini) that outperforms its standard counterpart on GPQA Diamond (+19.2 percentage points) underperforms on ISOSCI (-24.7 percentage points), showing that benchmark choice determines conclusions about reasoning utility. We release ISOSCI at this https URL
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.01431 [cs.CL] |
| (or arXiv:2607.01431v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.01431 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Samir Abdaljalil [view email]
[v1]
Wed, 1 Jul 2026 19:49:49 UTC (414 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.