ITPEval: Benchmarking Formal Translation Across Interactive Theorem Provers
Quick Answer
ITPEval introduces a benchmark for formal proof translation across Lean 4, Rocq, Isabelle, and HOL Light, featuring 1,560 source files and 6,848 theorems.
Quick Take
The benchmark reveals a peak statement translation pass rate of 29.1% and proof translation at 10.5%, highlighting library mismatch as a key bottleneck. The unified verification infrastructure is publicly available, enhancing multi-ITP context for improved success rates.
Key Points
- Benchmark includes 1,560 source files and 6,848 theorems across four ITPs.
- Statement translation achieves a peak pass rate of 29.1%; proof translation at 10.5%.
- Library mismatch identified as the dominant bottleneck in translation success.
- Publicly released verification infrastructure supports multi-ITP context improvements.
- Controlled theorems show a 29.7% proof pass rate versus 5.2% for ecosystem translations.
Paper Resources
Source Excerpt
Formal theorem proving has emerged as a frontier challenge for machine learning, yet the ecosystem is fragmented: proofs remain siloed across incompatible systems, limiting both training data for learning-based provers and the portability of verified results. We present ITPEval, the first benchmark for evaluating automated formal proof translation across four major ITPs (Lean 4, Rocq, Isabelle, and HOL Light), spanning two distinct logical foundations. Our benchmark comprises 1,560 source files
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →AINTMA: Agentic AI Architecture for Autonomous Test Management with Generative Intelligence, Secure Cloud Communication and Adaptive Quality Analytics
AINTMA, an autonomous test management architecture utilizing six specialized AI agents, achieves 88.4% test prioritization accuracy and reduces defect escape rates from 8.3% to 2.1%. The system demonstrates a 340% ROI within nine months, showcasing the potential of agentic AI in enhancing software quality management in cloud environments.