AdvancedMathBench: A Benchmark Suite for Advanced Mathematical Proof Generation and Verification

Lingkai Kong, Zijian Wu, Yuzhe Gu, Haiteng Zhao, Wenyong Huang, Shuang Sun, Zhicheng Xiong, Xiaotian Zhang, Shuya Zhao, Yan Wang, Disheng Xu, Wenwei Zhang, Kai Chen

AdvancedMathBench evaluates LLM reasoning through expert-aligned proof generation and process-level verification.

How do current LLMs perform on advanced, university-level mathematical proof generation and verification, and how can we reliably evaluate them?

Existing benchmarks for mathematical reasoning rely on final-answer correctness, which fails to distinguish between a rigorous proof and a lucky guess. This leaves the actual reasoning capabilities of models on advanced undergraduate and doctoral-level mathematics largely unmeasured. AdvancedMathBench introduces a two-part suite: ProverBench for generating natural-language proofs and VerifierBench for assessing the validity of those proofs. The authors use an expert-aligned automatic verification pipeline that combines rationale-aware reinforcement learning, positive-sample repair, and pessimistic verification to identify subtle logical failures. Frontier models struggle significantly with this benchmark, with the best-performing model achieving only 48.9% on doctoral-level problems and failing to reliably detect logical errors in plausible proof trajectories.

Paper Primer

The benchmark addresses the "plausibility trap," where models produce proofs that look correct but contain fatal logical gaps. By moving from outcome-based evaluation to process-level verification, the authors force models to justify every step rather than just arriving at a correct final value.

The core mechanism is an automatic verification pipeline: it uses a meta-verifier to reward models for producing expert-aligned rationales, augments training data with repaired proofs to prevent bias toward rejection, and applies pessimistic verification—accepting a proof only if all independent verification passes agree on its validity.

Advanced mathematical proof generation remains a major bottleneck for frontier LLMs.

On the doctoral qualifying-exam (QE) split of ProverBench, the top-performing model (GPT-5.5-xhigh) achieves only 48.9% accuracy. A significant performance drop is observed across all models when moving from undergraduate (UG) to QE-level problems.

Binary validity matching significantly overestimates model verification capabilities.

Moving from rough binary evaluation to meta-verification (which accounts for rationale quality) lowers the average Balanced F1 score by 9.0 points. Models often predict the correct validity polarity while providing incomplete or erroneous verification rationales.

Why is this benchmark necessary if we already have competition-level math benchmarks?

Existing benchmarks emphasize final-answer checking or coarse solution matching, which cannot determine if a proof is mathematically valid. AdvancedMathBench specifically targets the rigor of the reasoning process, which is essential for undergraduate and doctoral-level mathematics.

What is the primary failure mode of current models in this setting?

Models tend to over-accept plausible but invalid proof trajectories. They often struggle with conservative error detection, failing to identify subtle logical gaps that invalidate the global reasoning chain.

Researchers should shift from outcome-based metrics to process-level verification, as binary correctness is an insufficient proxy for mathematical rigor in advanced proof settings.

Introduction and Motivation

We expose the lack of advanced‑math evaluation and introduce AdvancedMathBench to fill it.

Current math benchmarks stop at competition‑level problems and evaluate only final answers, leaving the validity of intermediate reasoning unchecked. This leaves a blind spot for advanced university‑level proofs, where rigorous step‑by‑step justification is essential.

**Figure 1:** ADVANCEDMATHBENCH exposes complementary weaknesses in advanced mathematical proof generation and verification. Left: compared with answer-centric benchmark HMMT and competition-style proof benchmark USAMO, models on PROVERBENCH produce substantially lower scores, especially on the harder QE split. Right: on VERIFIERBENCH, models obtain low Balanced F1, indicating difficulty in proof verification.

To address this gap we introduce AdvancedMathBench, comprising ProverBench for proof generation and VerifierBench for proof verification. ProverBench offers 245 problems across undergraduate (UG) and doctoral qualifying‑exam (QE) levels, while VerifierBench supplies 888 model‑generated trajectories paired with expert annotations for fine‑grained error localization.

Experiments show that even the strongest frontier models remain far from human performance: GPT‑5.5‑xhigh attains only 64.5 on UG and 48.9 on QE, and the best verifier reaches a Balanced F1 of 65.1. The automatic verification pipeline, trained on expert data, achieves 82.1 Balanced F1 on held‑out examples, outperforming LLM‑as‑judge baselines.

The gap between competition math and university‑level proof verification remains large.

Mathematical Reasoning Benchmarks

We situate AdvancedMathBench among existing mathematical reasoning and proof verification benchmarks.

Mathematical reasoning benchmarks have progressed from simple arithmetic to full‑scale proof evaluation, motivating the need for a dataset that tests rigorous, checkable reasoning at the university level.

GSM8K is a collection of grade‑school word problems that require several reasoning steps, serving as a baseline for arithmetic and elementary algebra evaluation.

MATH aggregates competition‑level problems (algebra, geometry, counting) that demand multi‑step symbolic manipulation and logical inference.

Benchmarks olympiad‑level algebra, number theory, geometry, and combinatorics, pushing models beyond textbook problems toward contest difficulty.

A broad benchmark covering a wide spectrum of advanced mathematics, from undergraduate to graduate topics.

Open‑ended university‑level mathematics benchmark that emphasizes problem formulation and solution generation.

Graduate‑level applied derivations benchmark that stresses detailed symbolic manipulation.

Expert‑designed research‑level problems intended to probe the limits of current models.

A continuously updated platform that adds new mathematical reasoning tasks over time.

Early corpus for natural‑language proof generation, focusing on retrieval and proof completion.

Extends NaturalProofs with retrieval‑augmented proof completion and grounding in mathematical language.

Human‑evaluated collection of LLM‑generated proofs, revealing that correct answers do not guarantee valid proofs.

Framework that scores proofs along multiple dimensions: correctness, simplicity, readability, and methodological adaptation.

Large‑scale informal theorem‑proving benchmark that studies end‑to‑end proof generation at scale.

Benchmarks olympiad‑level problems with separate evaluation of answer, proof writing, and proof grading.

Step‑level human feedback dataset that demonstrates the value of process supervision over outcome‑only supervision.

Evaluates error‑step identification and first‑error localization in mathematical reasoning traces.

Open‑ended process verification benchmark that tests binary validity judgments and rationale generation for proofs.

Challenging verification tasks that stress LLM judges’ ability to assess nuanced mathematical arguments.

Analyzes the effectiveness of LLM‑generated critiques for improving mathematical reasoning performance.

Interactive proof assistant that provides machine‑checkable guarantees for formalized mathematics.

Higher‑order logic environment for constructing and verifying formal proofs.

Minimalist formal system that emphasizes proof transparency and compactness.

Interactive theorem prover with a powerful tactic language and growing mathematical library.

Benchmark Design and Splits

How ProverBench and VerifierBench are built, from source collection to expert‑validated splits.

The construction proceeds through a four‑stage curation pipeline that yields two complementary benchmarks: ProverBench for proof generation and VerifierBench for proof verification.

Problem Sourcing – gather problems from undergraduate exams, doctoral qualifying exams, contest archives, and textbook exercises.

Parsing & Filtering – convert PDFs to a unified format and discard answer‑centric question types.

Difficulty Pre‑screening – generate multiple LLM proofs, verify each repeatedly, and keep only items with high verifier uncertainty.

Expert Quality Control – PhD‑level mathematicians review statements and trajectories, removing ambiguous or trivial cases.

Split the curated set into ProverBench (proof generation) and VerifierBench (proof verification) with UG and QE difficulty partitions.

Compute the outcome frequencies for each LLM; the first LLM shows 3 accepts / 2 rejects → entropy ≈ 0.97 bits, indicating moderate uncertainty.

Low‑entropy problems (e.g., all accepts) are filtered out; this problem survives because its entropy exceeds the preset threshold of 0.8 bits.

Retained proofs are passed to human experts for final validation.

Uncertainty‑driven filtering preserves proofs that are plausible yet ambiguous, providing rich verification cases for VerifierBench.

ProverBench asks a model to write a complete, step‑by‑step natural‑language proof, much like drafting a detailed recipe rather than just naming the final dish.

How does ProverBench differ from earlier proof‑generation datasets like MATH?

MATH focuses on short answer‑only problems and evaluates via exact match, whereas ProverBench requires full, process‑level proofs and judges them with a verification pipeline that tolerates multiple correct reasoning paths.

VerifierBench treats verification like a meticulous fact‑checker who reads an entire proof, flags every mistake, and pinpoints the first fatal error.

Why does VerifierBench require a full‑chain annotation instead of stopping at the first error?

Because the earliest slip is often superficial; a later fatal mistake may be the true cause of invalidity. Full‑chain annotation captures all error signals, enabling models to learn to locate the decisive fault.

**Figure 2.** Overview of the AdvancedMathBench methodology. The schema summarizes problem sourcing and curation, the construction of ProverBench and VerifierBench, and the evaluation protocols connecting the automatic verification pipeline with the meta-verifier.

**Figure 3.** Subject distributions of ProverBench. The UG split of 200 samples covers core undergraduate mathematics, while the QE split of 45 samples emphasizes more advanced qualifying-exam-level topics.

Automatic Verification Pipeline

The section details the end‑to‑end automatic verification pipeline for ProverBench.

To evaluate model‑generated proofs at scale we construct an expert‑aligned verification pipeline that can handle diverse proof strategies, even when they diverge from the reference solution.

The pipeline turns raw model proofs into a vetted binary judgment by iteratively enriching training data, rewarding grounded analysis, and demanding unanimous agreement across multiple independent checks.

Generate multiple candidate proofs for each problem using LLMs (e.g., GPT‑5.5, DeepSeek‑V4‑Pro).

Apply the verifier‑uncertainty selector to pick trajectories that are most informative for annotation.

Experts annotate selected trajectories, producing ≈ 2 k labeled examples.

Repair rejected proofs via expert suggestions; verify repaired proofs with strong models and add ≈ 1.2 k positives.

Train the verifier with GRPO reinforcement learning, using the meta‑verifier’s graded reward (1.0, 0.5, 0.25, 0).

During inference, run eight independent verification passes in parallel; accept only if all eight agree.

Expert annotates P₁ as negative; the system repairs it using the expert’s correction hint, producing P₁ʹ.

P₁ʹ passes verification by the meta‑verifier with a `BASIC_MATCH` (reward 0.5) and is added to the positive pool.

During evaluation, the trained verifier runs eight parallel checks on P₃; all eight return “accept”, so the proof is marked valid.

The repair step converts a false negative into a useful training signal, while the eight‑pass unanimity rule dramatically lowers the chance of a subtle error slipping through.

How does this verification pipeline differ from a standard binary classifier that simply predicts “correct” or “incorrect”?

Standard classifiers learn a static decision boundary from a fixed dataset. Our pipeline continuously enriches its training set with repaired proofs, uses a graded meta‑verifier reward to encourage nuanced reasoning, and requires unanimous agreement across multiple independent verification passes, which together yield far stronger robustness to novel proof styles.

On a held‑out set of 94 examples the verifier attains a Rough F1 of 82.1 % versus the GPT‑5.5‑xhigh baseline’s 70.6 %, demonstrating the practical benefit of the full pipeline.

Experimental Results

Key results expose a large gap between proprietary and open‑source models on advanced proofs.

We evaluate a broad set of top‑tier LLMs under a uniform protocol: temperature 1.0, 64 k‑token limit, and maximal reasoning effort.

The table presents a comparison of various AI models categorized into "Proprietary Models" and "Open-source Models" across two metrics, "UG" and "QE".

Proprietary models outperform open‑source models by $24.5$ points on the UG split of ProverBench.

Best proprietary UG score $64.5$ vs best open‑source UG score $40.0$ (Table 1).

**Table 2.** Main results on VERIFIERBENCH. Rough scores evaluate only the validity polarity, whereas Meta-Ver scores additionally evaluate agreement with expert annotations in verification rationale and error localization. All scores are percentages.

**Table 3.** Ablations on Automatic Verification Pipeline

Meta‑Ver Score measures how well a model’s verification rationale matches expert annotations, not just whether it says “valid” or “invalid”.

The performance gap between proprietary and open‑source models remains large on advanced proof tasks, highlighting the need for stronger open‑source research.

Benchmark Subject Distributions

Appendix A details the subject breakdowns and example problems that illustrate the benchmark.

Appendix A presents the subject frequency tables for the undergraduate (UG) and qualifying‑exam (QE) splits of ProverBench, then gives representative proof problems from ProverBench and VerifierBench.

The table lists 11 mathematical subjects ranked by their frequency count and corresponding percentage.

The table presents a ranking of mathematical subjects based on their frequency count and corresponding percentage.

ProverBench example (UG‑Advanced Algebra): prove that for $n\times n$ positive‑semidefinite real symmetric matrices $A$ and $B$, $AB=0$ iff $\operatorname{tr}(AB)=0$.

ProverBench example (QE‑Analysis): show that any tempered distribution $u\in\mathcal{S}'(\mathbb{R}^n)$ supported at $\{0\}$ can be written as $u(\varphi)=\sum_{|\alpha|\le N}c_\alpha\,\partial^\alpha_x\varphi(0)$ for all $\varphi\in\mathcal{S}(\mathbb{R}^n)$.

VerifierBench example formulates a variational problem: among smooth simple closed curves $C$ with fixed enclosed area $A$, the circle minimizes perimeter $L$, leading to the isoperimetric inequality $L^2\ge 4\pi A$ with equality only for circles.

Expert annotation flags a fatal error in Step 9: the proof treats the stationary condition as a global minimum without first establishing existence of a minimizer, making the argument incomplete.

Prompting Strategy

Appendix B lists the prompts used for proof generation, verification, and meta‑verification.

This appendix records the exact prompts that drive the three benchmark tasks.

Annotation Interface

Details the annotation protocol and error taxonomy for proof evaluation.

The annotation protocol presents annotators with a problem, a model‑generated proof, and auxiliary traces, but requires them to rely on their own mathematical judgment.

The table defines fields for annotating mathematical proofs, including "Reviewer Comment," "Remark," "First Fatal Error Step," "Recoverable Error Steps," and "Report Error Case."

Read the original paper

Open the simplified reader on Paperglide

Browse all simplified papers