{
 "body": "\n## What it measures\n\nProofBench asks for a Lean 4 proof that compiles, not a contest write-up. Each of the 100 scored items gives a natural-language theorem and a vetted Lean statement. The model may search Mathlib and execute Lean, then it must submit one proof. The kernel accepts the proof or the item is wrong.\n\nProblems are pitched at advanced undergraduate and graduate level, from qualifying exams and textbooks, covering analysis, algebra, probability, number theory, and logic. Epoch AI mirrors the Vals board and describes the same private split and tool loop.\n\n## How it is scored\n\nScoring is binary proof success rate, as a percentage of 100. No partial credit. Vals v1.1 (HTML updated 4 September 2026) lists Logical Intelligence AlephProver at 100%, Claude Opus 5 at 99%, Claude Fable 5 at 95%, Kimi K3 at 87%, Harmonic Aristotle at 86%, and GPT-5.6 Sol at 83%. Vals notes that a one-problem gap is not evidence, and that a perfect score is a ceiling. Models that never call `submit_proof` score near zero even if early Lean steps looked fine (Vals cites DeepSeek V4 submitting 16/100).\n\nThe FormalProofBench paper (27 March 2026) reported 33.5% for the best foundation model in that snapshot. That number is not the September 2026 board.\n\n## Dataset and licence\n\nThe leaderboard uses 100 held-out problems. `vals-ai/proof-bench` says the `problems/` tree holds samples only. Vals tags the product proprietary; a single SPDX id for the private set was not found. Test answers are not public. English informal statements plus Lean 4 code.\n\n## Who publishes it\n\nVals AI, with Lean 4 experts. Paper: Nikil Ravi, Kexing Ying, Vasilii Nesterov, Rayan Krishnan, Elif Uskuplu, Bingyu Xia, Janitha Aswedige, Langston Nashold, arXiv 2603.26996, ICLR 2026 workshop VerifAI-2. Live board: `vals.ai/benchmarks/proof_bench`. Epoch AI republishes those scores and does not run a second exam.\n\n## Lineage\n\nThis page is the Vals/Epoch Lean 4 ProofBench, including the FormalProofBench paper name and the v1.1 board. It is not the Berkeley ProofBench used to train ProofGrader (arXiv 2510.13888, 145 contest problems, 0\u20137 expert grades on natural-language proofs). It is not [Putnam-AXIOM](putnam_axiom.md), which scores boxed answers. MiniF2F and PutnamBench have no pages here.\n\n## Saturation and contamination\n\nOn the 4 September 2026 v1.1 table the top is at 100% and the next at 99%, so the 100-problem split no longer separates the leaders. Lower ranks still move with tool use and submission rate. The scored set is private, so leakage risk is low relative to public contest archives. Textbook overlap remains possible.\n\n## How to run it\n\nPublic samples: `python main.py --dataset exported --model ... --k 3` in `vals-ai/proof-bench` after `SETUP.md`. That run is not the private 100. Live numbers come from Vals, with up to 40 turns and the three-tool bundle. Aristotle uses its own harness (Epoch stores it, does not chart it). No lm-eval task was found. Do not mix a k-sample local run with the hosted board.\n\n## Reading the numbers\n\n100% means every private item compiled, not that the model can formalise arbitrary research math. A 99 vs 100 gap is one problem. Cost and latency on the Vals page vary by more than the accuracy spread (Vals quotes $9.35 per task for AlephProver vs $1.79 for Claude Opus 5). A low score can be a submission-loop failure rather than a math failure. Compare against a natural-language proof set only as a different skill.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "proofbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "FormalProofBench",
   "Proof Bench",
   "ProofBench v1.1"
  ],
  "category": "math",
  "contamination": {
   "note": "The scored split is private. Sample Lean files in the public repo are not the board. Lean 4 checking blocks memorised natural-language write-ups. Textbook-origin statements could still overlap public formalisations.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": false,
   "size": 100,
   "size_note": "Leaderboard split is 100 private problems. The GitHub repo ships only sample Lean files; remaining statements are held out. Epoch describes the same 100-problem private split. Textbook/qualifying-exam sources; domains include analysis, algebra, probability, number theory, and logic.\n",
   "splits": "private 100-problem leaderboard split; sample problems in the repo",
   "url": "https://vals.ai/benchmarks/proof_bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-079 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "github.com/vals-ai/proof-bench: python main.py --dataset exported --model ... --k 3 Tools: lean_run_code, lean_loogle, submit_proof. Epoch: up to 40 turns, Mathlib search plus code execution. Aristotle uses a separate harness.\n"
  },
  "id": "proofbench",
  "last_updated": "2026-09",
  "leaderboard_url": "https://vals.ai/benchmarks/proof_bench",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ProofBench (Vals AI; paper title FormalProofBench) tests whether a model can turn graduate-level math into a machine-checked Lean 4 proof. Each item pairs an English problem with a vetted Lean statement. The model may search Mathlib and run Lean, then must submit one proof. The checker either accepts it or not. There is no partial credit for a plausible write-up. Problems come from qualifying exams and textbooks across analysis, algebra, probability, number theory, and logic, not from contest short-answer keys.\n",
  "metric": {
   "baseline_note": "Binary per problem. Vals ProofBench v1.1 (updated 4 Sep 2026): AlephProver 100%, Claude Opus 5 99%, Claude Fable 5 95%, Kimi K3 87%, Harmonic Aristotle 86%, GPT-5.6 Sol 83% on the 100-problem private split. The March 2026 FormalProofBench paper reported 33.5% for the best foundation model then tested.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "proof success rate",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "ProofBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2603.26996",
   "title": "FormalProofBench: Can Models Write Graduate Level Math Proofs That Are Formally Verified?",
   "url": "https://arxiv.org/abs/2603.26996",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Nikil Ravi",
    "Kexing Ying",
    "Vasilii Nesterov",
    "Rayan Krishnan",
    "Elif Uskuplu",
    "Bingyu Xia",
    "Janitha Aswedige",
    "Langston Nashold"
   ],
   "org": "Vals AI",
   "url": "https://vals.ai/benchmarks/proof_bench"
  },
  "released": "2026-03",
  "repo_url": "https://github.com/vals-ai/proof-bench",
  "saturation": {
   "as_of": "2026-09",
   "note": "Vals v1.1 board dated 4 Sep 2026 lists AlephProver at 100% and Claude Opus 5 at 99%. Vals itself warns that one problem in 100 is not a real gap and that a perfect score is a ceiling. The March 2026 workshop paper's 33.5% figure is a different snapshot, not a live number.\n",
   "status": "saturated",
   "top_score": 100
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Vals AI ProofBench v1.1 leaderboard (updated 4 Sep 2026)",
    "url": "https://vals.ai/benchmarks/proof_bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "Epoch AI ProofBench methodology (Vals source, 100 private, 40 turns)",
    "url": "https://epoch.ai/benchmarks/proofbench"
   },
   {
    "accessed": "2026-09-08",
    "title": "FormalProofBench arXiv abs (submitted 27 Mar 2026; ICLR 2026 workshop)",
    "url": "https://arxiv.org/abs/2603.26996"
   },
   {
    "accessed": "2026-09-08",
    "title": "vals-ai/proof-bench README (tools and exported dataset flag)",
    "url": "https://raw.githubusercontent.com/vals-ai/proof-bench/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "vals-ai/proof-bench repository",
    "url": "https://github.com/vals-ai/proof-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "ProofGrader / academic ProofBench (name collision; NL proofs)",
    "url": "https://arxiv.org/abs/2510.13888"
   }
  ],
  "status": "active",
  "subcategory": "Lean 4 graduate-level formal theorem proving",
  "summary": "Private 100-problem Lean 4 suite: given a natural-language statement and a formal theorem, the model must write a proof the checker accepts.\n",
  "tags": [
   "math",
   "lean",
   "formal-verification",
   "agentic",
   "private-test"
  ],
  "task_format": "Agentic loop with lean_loogle, lean_run_code, and a single submit_proof. Up to 40 turns (Epoch and Vals write-ups). Natural-language statement plus Lean 4 theorem in; compiled proof out.\n"
 }
}