Hallucinations on the Board: Tool-Augmented Evaluation of LLM Chess Commentary
Quick Answer
The ACT-Eval framework evaluates chess commentary from LLMs, revealing that GPT-5.4 produces incorrect sub-claims 22% of the time, while smaller models exceed 40%.
Quick Take
Tool augmentation improves factual correctness but fails to cover expert strategic ideas adequately, impacting both experts and novices.
Key Points
- ACT-Eval decomposes chess commentary into atomic claims for evaluation.
- Benchmark includes 325 position-move pairs with expert-verified gold atoms.
- Tool augmentation significantly improves factual correctness and move assessment.
- Factual hallucinations are common, with smaller models exceeding 40% incorrect claims.
- Human calibration shows strong correlation with expert assessments of strategic completeness.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Superhuman game engines in domains like chess have made expert-level evaluations easily accessible, yet they communicate what is true without the natural-language explanations that make such expertise educationally useful to experts and non-experts alike. Large language models could, in principle, bridge this gap, but they frequently hallucinate due to limited domain-specific knowledge, and standard reference-based or LLM-as-a-judge frameworks cannot reliably detect these errors. In this work, we present ACT-Eval, an evaluation framework that decomposes chess commentary into atomic claims and routes them to engine-supported tools and expert-annotated gold references to assess factual correctness, conceptual coverage, and move-quality judgment. We release a benchmark of 325 position--move pairs spanning pedagogical, tournament, and critical positions, including 125 positions with expert-verified gold atoms and a five-class error taxonomy. Evaluating leading proprietary and open-weight models, we find that factual hallucinations remain pervasive in chess commentary: GPT-5.4 without tools produces incorrect sub-claims 22.0% of the time, while smaller open-weight models exceed 40%. Although tool augmentation substantially improves factual correctness and move-quality assessment, coverage of expert strategic and tactical ideas remains limited across all models. Human calibration shows that ACT-Eval's factual judgments fall within the observed range of inter-human agreement, while its coverage scores correlate strongly with human assessments of strategic completeness.
| Comments: | 23 pages, 6 figures |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2608.04240 [cs.CL] |
| (or arXiv:2608.04240v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2608.04240 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: S Ashwin Hebbar [view email]
[v1]
Tue, 4 Aug 2026 21:31:01 UTC (1,272 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.