Cost-Effective Automated Judging of Natural-Language Mathematical Proofs
Quick Answer
This paper shows that Cost-effective models like GPT-OSS 120B and DeepSeek-V4 Flash achieve human-level accuracy in grading mathematical proofs, matching elite models like Claude Opus 4.7 at a fraction of the cost.
Quick Take
A unanimous agreement rule (all-three-pass) maximizes grading precision, demonstrating that cheaper judges can compete effectively in this domain.
Key Points
- Three low-cost models achieve human agreement rates similar to expensive models.
- Models tested include GPT-OSS 120B, DeepSeek-V4 Flash, and Gemma-4 31B.
- Unanimous agreement (all-three-pass) yields highest grading precision.
- Cost savings are up to 100 times lower than frontier models.
- Study based on a 200-instance validation sample from IMO-GradingBench.
DeepSignal Analysis
What happened
Recent research indicates that cost-effective models like GPT-OSS 120B and DeepSeek-V4 Flash can achieve grading accuracy comparable to more expensive models such as Claude Opus 4.7. The study utilized a validation sample from IMO-GradingBench to assess the performance of these models in grading mathematical proofs.
Key evidence
- Three inexpensive models, including GPT-OSS 120B and DeepSeek-V4 Flash, matched human grading decisions at rates statistically similar to Claude Opus 4.7.
- The cost of using these cheaper models can be up to 100 times lower than that of elite models like Gemini 3.1 Pro.
- The study found that requiring unanimous agreement among the three models maximized grading precision and minimized variability across runs.
Why it matters
The findings suggest that organizations can leverage cheaper AI models for grading mathematical proofs without sacrificing accuracy. This could lead to significant cost savings in educational and research settings where large volumes of proofs need evaluation. However, the recommendation for unanimous agreement as a grading strategy requires further validation.
What to watch
Paper Resources
Source Excerpt
Grading natural-language mathematical proofs is a recurring cost in evaluating math-reasoning systems, and frontier judges are expensive. We ask whether cheap open-weight models can serve as reliable judges given a candidate proof, a ground-truth proof, and a human-grading rubric. On a 200-instance validation sample of IMO-GradingBench, three cheap judges (GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B) agree with human pass/fail decisions at rates statistically indistinguishable from Claude O
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.