Evaluating SageMath-Augmented LLM Agents for Computational and Experimental Mathematics
Quick Answer
The study evaluates a ReAct-style agentic setup combining LLMs with SageMath, achieving an average performance gain of 9.7 percentage points on the RealMath benchmark.
Quick Take
The study evaluates a ReAct-style agentic setup combining LLMs with SageMath, achieving an average performance gain of 9.7 percentage points on the RealMath benchmark. Qwen 3.7-Max shows the highest benefit from SageMath, while GPT-5.5 achieves a 75.2% solve rate, indicating a promising direction for automated conjecture discovery in computational mathematics.
Key Points
- SageMath integration leads to substantial performance gains across evaluated models.
- Qwen 3.7-Max benefits the most from SageMath access.
- GPT-5.5 achieves the highest solve rate of 75.2% among tool-enabled configurations.
- A multi-step post-processing and validation pipeline enhances problem set quality.
- The project aims to assist mathematicians in computational exploration.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Recent advances in AI for Mathematics have focused largely on autoformalization and theorem proving, leaving the role of Computer Algebra Systems (CAS) in agentic LLM workflows underexplored. We propose a ReAct-style agentic setup that combines LLM reasoning with verifiable feedback from SageMath, together with Context7 for the up-to-date documentation. We evaluate this agentic setup across frontier models for solving research-level mathematical problems from the RealMath benchmark in a setting that emulates a computational-mathematics research loop. We also propose a refinement to the RealMath benchmark by introducing a multi-step post-processing procedure and a multi-stage validation pipeline, both of which improve the quality and reliability of the extracted problem set. Our experiments reveal substantial performance gains from SageMath access across all evaluated models on +9.7~pp on average, the gains range from 1.5~pp to 27.8~pp and narrow the gap between open-weight and closed models. Qwen~3.7-Max benefits from SageMath the most, while GPT-5.5 achieves the highest solve rate of $75.2\%$ and the lowest token usage among tool-enabled configurations. Our findings suggest that CAS-augmented agents represent a promising direction for assisting mathematicians in computational exploration, and we believe that this work is a step towards automated conjecture discovery. The project repository is available online.
| Comments: | 37 pages, 16 figures, accepted to 3rd AI for Math Workshop at ICML 2026 |
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.06820 [cs.AI] |
| (or arXiv:2607.06820v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.06820 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Pavel Snopov [view email]
[v1]
Tue, 7 Jul 2026 21:29:59 UTC (6,560 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.