PHREEQC-MCQ-200: A Diagnostic Benchmark for Tool-Augmented Scientific Simulator Agents
Quick Answer
PHREEQC-MCQ-200 is a benchmark for evaluating tool-augmented agents in aqueous-geochemistry simulations, revealing that simulator access enhances accuracy but can also lead to regressions.
Quick Take
The study emphasizes the importance of evaluating scientific agents not just on accuracy but also on retention and output-access sensitivity.
Key Points
- Benchmark consists of 200 multiple-choice questions from 21 validated PHREEQC scenarios.
- Simulator access significantly improves aggregate accuracy across various model families.
- Tool-augmented agents can regress on items they previously answered correctly without tools.
- Output-access protocol impacts performance, with table-of-contents interfaces benefiting stronger models.
- Evaluations should report accuracy, item-level retention, and computation chain failures.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Large language model agents are increasingly connected to scientific software, yet it remains unclear when tool access makes scientific computation more reliable rather than merely more complex. We introduce PHREEQC-MCQ-200, a benchmark for evaluating tool-augmented agents on deterministic aqueous-geochemistry simulations. The benchmark contains 200 multiple-choice questions derived from 21 validated PHREEQC scenarios, requiring agents to construct simulator inputs, execute PHREEQC, inspect structured outputs, and commit to final answers.
Across multiple frontier and mid-tier model families, simulator access substantially improves aggregate accuracy, confirming that grounded execution is necessary for many scientific-computation tasks. However, the gains are not monotonic: tool-augmented agents also lose items they answered correctly without tools, revealing regressions that average accuracy alone hides. We further show that output-access protocol matters. A table-of-contents interface can reduce token cost while preserving or improving accuracy for stronger models, but it degrades performance for mid-tier models that cannot reliably navigate structured simulator outputs.
PHREEQC-MCQ-200 therefore frames scientific tool use as an end-to-end diagnostic problem rather than a simple tool-calling capability. We argue that evaluations of scientific agents should report not only accuracy, but also item-level retention, output-access sensitivity, trajectory failures, and where the computation chain breaks.
| Comments: | 30 pages, 2 figures |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.00436 [cs.AI] |
| (or arXiv:2607.00436v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.00436 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ke Zhang [view email]
[v1]
Wed, 1 Jul 2026 04:51:37 UTC (3,487 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.