{
 "body": "\n## What it measures\n\nMATH gives a model a written competition mathematics problem -- drawn from AMC, AIME and similar\ncontests, spanning algebra, geometry, number theory, counting and probability, precalculus,\nintermediate algebra and prealgebra, each labelled with a difficulty level from 1 (easiest) to 5\n(hardest) -- and asks for a full step-by-step solution ending in a final answer, conventionally\ninside `\\boxed{}`. There are no answer choices. It tests the same skill as MATH-500 (this\nrepository's `math_500` page, a fixed 500-problem subset of MATH's test split): multi-step symbolic\nand numeric reasoning. The distinction that matters for reading model cards is size and provenance,\nnot task design -- MATH and MATH-500 test the identical kind of problem, just at 5,000 versus 500\nitems.\n\n## How it is scored\n\nGrading compares a model's extracted final answer against a reference answer, typically the text\ninside the last `\\boxed{}` or `\\fbox{}` in the response (this logic, from the original repository, is\nreused verbatim in HELM's `math_scenario.py`). Because a correct answer can be written more than one\nway (`1/2` versus `0.5`), some implementations -- including OpenAI's simple-evals -- use an LLM\nequality checker rather than exact string match; others use symbolic comparison via SymPy (the\nMinerva-derived harness variant). Reported numbers are usually pass@1 accuracy, but papers differ on\nprompt format, shot count, and sample averaging, so two \"MATH\" numbers are only comparable once both\ngraders and both test sets match. The paper's own low-capacity baseline, a fine-tuned GPT-2, scored\n6.9%; no confirmed human baseline was published.\n\n## Dataset and licence\n\nMATH is 12,500 problems: 7,500 for training and 5,000 for test, each with a full worked solution, a\nsubject label and a 1-5 difficulty level, confirmed by summing the seven per-subject splits on\ncurrent mirrors. The original GitHub repository (`hendrycks/math`) states an MIT licence, also\ncarried on its Hugging Face mirrors. The canonical upload, `hendrycks/competition_math`, was disabled\nin January 2025 following a DMCA takedown by Art of Problem Solving, which alleged more than 10,000\nof the 12,500 problems were copied from its Alcumus platform (the same notice affects MATH-500's\nprovenance, documented on that page). The repository now points users to a replacement mirror,\n`qwedsacf/competition_math` -- which collapses the dataset into one undivided 12,500-row split rather\nthan preserving the train/test boundary, a trap for anyone assuming that split still applies there.\n`DigitalLearningGmbH/MATH-lighteval`, adopted by Inspect Evals after the takedown, does preserve the\noriginal 7,500/5,000 split.\n\n## Who publishes it\n\nMATH was introduced by Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric\nTang, Dawn Song and Jacob Steinhardt at UC Berkeley, published at NeurIPS 2021 (Datasets and\nBenchmarks track). No organisation runs a live public leaderboard for the full MATH test; Hendrycks\nand coauthors maintain the reference repository, and downstream harnesses (lm-evaluation-harness,\nHELM, Inspect Evals, OpenCompass) each maintain their own copy of the evaluation logic.\n\n## Lineage\n\nMATH has no predecessor of its own; it was built to go beyond grade-school arithmetic benchmarks like\nGSM8K. Two things came after it that matter for this repository. First, OpenAI carved out MATH-500\n(`math_500`) in 2023 by holding out 500 test problems as a fixed evaluation slice and moving the\nother 4,500 into training data for their own process-supervision work -- MATH-500, not the full MATH\ntest, is what most current model cards and system cards report under a \"MATH\" column, which is\nexactly the source of confusion this page exists to flag. Second, Omni-MATH (`omni_math`, also\ndocumented in this batch) was built in 2024 because MATH -- alongside GSM8K -- had become \"solved\nwith high accuracy,\" citing OpenAI o1's 94.8% on MATH as evidence, and targets olympiad-level\ndifficulty instead.\n\n## Saturation and contamination\n\nFull 5,000-problem MATH test scores for current frontier models are hard to find, because almost\nevery lab now reports MATH-500 instead (saturated at GPT-5 (high) 99.4% as of 2026-09, per that\npage). The clearest sourced trajectory on the full test located here: 6.9% for a fine-tuned GPT-2 at\nthe original 2021 publication, rising to 50.3% for Minerva 540B using majority voting over 256\nsamples (Lewkowycz et al., 2022) -- both predate MATH-500's introduction, so they are unambiguously\nfull-test numbers. No full-test score for a 2024-2026 model was found; saturation status for the full\ntest is therefore not established here, which is itself informative: if a recent paper reports a\nbare \"MATH\" number, check which split it used before treating it as comparable to another. Full-test\ncontamination risk is high given the DMCA dispute over the source problems and public circulation\nsince 2021.\n\n## How to run it\n\nlm-evaluation-harness exposes the full test as the `hendrycks_math` group (seven subject tasks,\nboxed-answer extraction) and separately as `minerva_math` (Minerva's 4-shot prompt and SymPy-based\nchecking); both groups also, confusingly, ship a same-family `..._math500` task that loads\n`HuggingFaceH4/MATH-500` instead -- so the harness task family name alone does not tell you which\nsplit ran. HELM's `math` scenario and Inspect Evals' `math` task (now backed by\n`DigitalLearningGmbH/MATH-lighteval` post-takedown) both cover the full test; OpenCompass ships dozens\nof `math_*` configs, including full-test variants and separate MATH-500-specific ones\n(`math_500_gen.py`, `math_prm800k_500_*.py`) in the same directory. Always confirm the dataset path\nor config name behind a \"MATH\" number rather than assuming it covers all 5,000 test problems.\n\n## Reading the numbers\n\nA MATH score only means what its test set means: 5,000 problems is a materially harder bar to sustain\na high score across than the 500-problem MATH-500 slice most current reports actually use, so do not\ntreat a \"MATH\" figure from a 2025-2026 model card as comparable to one from a 2021-2022 paper without\nchecking the split. Given the source dataset's contested licensing status, a high score is weak\nevidence of contamination-free reasoning either way. For the number most current model cards carry,\nsee this repository's `math_500` page; for a benchmark built to still separate frontier models, see\n`omni_math`.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "math",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Hendrycks MATH",
   "MATH dataset",
   "hendrycks_math",
   "minerva_math"
  ],
  "category": "math",
  "contamination": {
   "note": "The source dataset repository, hendrycks/competition_math, was disabled on Hugging Face following a January 2025 DMCA takedown filed by Art of Problem Solving, alleging over 10,000 of the 12,500 problems were copied from its Alcumus platform (the same notice covers MATH-500, documented on this repository's math_500 page). Both Inspect Evals and several re-uploads (for example qwedsacf/competition_math) now point to substitute mirrors; one popular replacement, qwedsacf/competition_math, collapses all 12,500 problems into a single undivided split rather than preserving the original train/test boundary, a real trap for anyone assuming a \"train\" or \"test\" split still applies there. DigitalLearningGmbH/MATH-lighteval, adopted by Inspect Evals as its replacement source after the takedown, does preserve the original 7,500/5,000 split.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 5000,
   "size_note": "12,500 problems total: 7,500 training and 5,000 test, confirmed by summing the seven per-subject splits on the EleutherAI/hendrycks_math and DigitalLearningGmbH/MATH-lighteval mirrors (the seven train subjects sum to exactly 7,500; the seven test subjects sum to exactly 5,000). MATH-500 (this repository's math_500 page) is 500 problems held out from this 5,000-problem test split.\n",
   "splits": "train (7,500), test (5,000), across seven subject subsets (algebra, counting_and_probability, geometry, intermediate_algebra, number_theory, prealgebra, precalculus)",
   "url": "https://github.com/hendrycks/math"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "math",
   "inspect_evals": "math",
   "lm_eval": "hendrycks_math (group of 7 subject tasks, plus a same-directory hendrycks_math500 task loading MATH-500 instead), minerva_math (Minerva's 4-shot prompt and SymPy answer checking, also with its own minerva_math500 task)",
   "opencompass": "math (many config variants, including full-test math_gen*.py configs and separate math_500_gen.py / math_prm800k_500_*.py configs in the same directory)",
   "other": "OpenAI's simple-evals math_eval.py, the source of the boxed-answer extraction logic HELM reuses, evaluates MATH-500 rather than the full test set."
  },
  "id": "math",
  "last_updated": "2025-01",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "omni_math"
   ],
   "variants": [
    "math_500"
   ]
  },
  "measures": "MATH gives a model a written competition mathematics problem -- spanning algebra, geometry, number theory, counting and probability, precalculus, intermediate algebra and prealgebra, each labelled with a difficulty level from 1 to 5 -- and asks for a full worked solution ending in a final answer. There are no answer choices. It tests the same skill as MATH-500 (this repository's `math_500` page, a fixed 500-problem subset of MATH's test split): multi-step symbolic and numeric reasoning, not speed, tool use, or any language beyond English. What differs between MATH and MATH-500 is size and provenance, not task design -- they test the identical kind of problem at 5,000 versus 500 items.\n",
  "metric": {
   "baseline_note": "The original paper's own baseline on the full test set was very low: a fine-tuned GPT-2 scored 6.9%. No confirmed human baseline was published for the full test set; free-response grading has no meaningful chance rate.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (pass@1, boxed-answer match)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MATH (Mathematics Aptitude Test of Heuristics)",
  "page_kind": "family",
  "paper": {
   "arxiv": "2103.03874",
   "title": "Measuring Mathematical Problem Solving With the MATH Dataset",
   "url": "https://arxiv.org/abs/2103.03874",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Dan Hendrycks",
    "Collin Burns",
    "Saurav Kadavath",
    "Akul Arora",
    "Steven Basart",
    "Eric Tang",
    "Dawn Song",
    "Jacob Steinhardt"
   ],
   "org": "UC Berkeley",
   "url": "https://github.com/hendrycks/math"
  },
  "released": "2021-03",
  "repo_url": "https://github.com/hendrycks/math",
  "saturation": {
   "as_of": "",
   "note": "Full 5,000-problem MATH test scores are hard to find for current frontier models because almost every lab now reports the 500-problem MATH-500 subset instead (see this repository's math_500 page, which is saturated -- GPT-5 (high) at 99.4% as of 2026-09). The clearest sourced full-test trajectory this research found: 6.9% (fine-tuned GPT-2, the original 2021 paper's own baseline) rising to 50.3% (Minerva 540B with majority voting over 256 samples, Lewkowycz et al. 2022); both figures predate MATH-500's introduction, so they are unambiguously full-test numbers. No full-test score for a 2024-2026 model was found during this research; if a paper reports a bare \"MATH\" number, verify which split it used before comparing it to another reported \"MATH\" number.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Measuring Mathematical Problem Solving With the MATH Dataset (Hendrycks et al., 2021)",
    "url": "https://arxiv.org/abs/2103.03874"
   },
   {
    "accessed": "2026-09-08",
    "title": "hendrycks/math GitHub repository (MIT LICENSE, README pointing to replacement mirror after takedown)",
    "url": "https://github.com/hendrycks/math"
   },
   {
    "accessed": "2026-09-08",
    "title": "hendrycks/competition_math dataset card (disabled following DMCA takedown)",
    "url": "https://huggingface.co/datasets/hendrycks/competition_math"
   },
   {
    "accessed": "2026-09-08",
    "title": "qwedsacf/competition_math dataset card (replacement mirror, single undivided split)",
    "url": "https://huggingface.co/datasets/qwedsacf/competition_math"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/hendrycks_math dataset (per-subject train/test splits)",
    "url": "https://huggingface.co/datasets/EleutherAI/hendrycks_math"
   },
   {
    "accessed": "2026-09-08",
    "title": "DigitalLearningGmbH/MATH-lighteval dataset card (Inspect Evals' post-takedown data source)",
    "url": "https://huggingface.co/datasets/DigitalLearningGmbH/MATH-lighteval"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals math task README (documents the January 2025 DMCA takedown)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/math"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM math_scenario.py",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/math_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass math dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/math"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness hendrycks_math task (includes hendrycks_math500.yaml)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/hendrycks_math"
   },
   {
    "accessed": "2026-09-08",
    "title": "Solving Quantitative Reasoning Problems with Language Models (Minerva paper, Lewkowycz et al., 2022) -- full MATH test score",
    "url": "https://ar5iv.labs.arxiv.org/html/2206.14858"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face legal takedown notice, Art of Problem Solving vs. hendrycks/competition_math",
    "url": "https://huggingface.co/datasets/huggingface-legal/takedown-notices/blob/main/2025/2025-01-02-AoPS.md"
   }
  ],
  "status": "active",
  "subcategory": "competition mathematics",
  "summary": "The original 12,500-problem competition-mathematics benchmark from Hendrycks et al. 2021; its 5,000-problem test split is the parent of the smaller MATH-500 subset most current model cards actually report.",
  "tags": [
   "math",
   "competition-math",
   "chain-of-thought",
   "free-response",
   "family"
  ],
  "task_format": "Free-response: read a competition mathematics problem, produce a worked solution and a final answer, conventionally inside \\boxed{}."
 }
}