New benchmark reveals AI models struggle with graduate-level math proofs, with top performers achieving only 33.5% accuracy on formally verified problems.

FormalProofBench introduces a private benchmark to test whether AI models can generate formally verified mathematical proofs at graduate level using Lean 4

Best-performing frontier models achieve 33.5% accuracy, with performance dropping significantly beyond the top contenders

Problems span advanced undergraduate and graduate mathematics including analysis, algebra, probability, and logic, sourced from qualifying exams and textbooks

Evaluation includes empirical analysis of tool-use patterns, failure modes, computational costs, and latency across multiple frontier models