{
 "body": "\n## What it measures\n\nOmni-MATH gives a model an olympiad-level mathematics problem drawn from international, national and\nregional competitions -- the IMO, Putnam, USAMO, HMMT and others -- and asks for a full solution and\nfinal answer, with no answer choices offered. Problems are organised into more than 33 sub-domains\n(algebra, number theory, geometry, combinatorics, calculus and more) and assigned one of ten\ndifficulty levels, calibrated primarily against the Art of Problem Solving community's own difficulty\nratings, with GPT-4o assigning a level to any problem AoPS had not already rated. It is built to\ndemand genuine olympiad-level reasoning rather than the standard competition mathematics that GSM8K\nand the original MATH dataset test, which the authors argue were already solved at high accuracy by\n2024-era models -- their own headline example is OpenAI o1 scoring 94.8% on MATH.\n\n## How it is scored\n\nThe official protocol extracts a model's final answer and checks it against the reference answer with\nGPT-4o as an equivalence judge, following a documented few-shot prompt; because free-form\nmathematical answers can be written in more than one equivalent form, exact string matching is not\nused. Because GPT-4o judging is expensive at scale, the authors separately released \"Omni-Judge,\" a\nLlama-3-8B-Instruct model fine-tuned on 21,451 of their own GPT-4o judgments, which they report\nagrees with GPT-4o about 95% of the time on a held-out internal test set of 2,690 examples. A\nrule-based evaluator, adapted from Qwen2.5-MATH's grading code and covering a subset of problems\nsuited to exact-match checking, was released in December 2024 as a third, cheaper option, which the\nauthors report is \"generally consistent\" with the GPT-4o-judged leaderboard. Because three different\ngraders exist, two reported scores are only safely comparable once you confirm they used the same\none.\n\n## Dataset and licence\n\nThe Hugging Face dataset (`KbsdJames/Omni-MATH`, Apache-2.0) ships 4,428 problems in a single public\n\"test\" split, each with a domain path, a difficulty rating, the problem statement, a full worked\nsolution and a final answer; no separate training split is published. Problems without a pre-existing\nAoPS solution were converted to LaTeX and difficulty-rated by the authors' own pipeline rather than\nsourced verbatim from a single existing corpus, so licensing follows the authors' Apache-2.0 grant on\nthe packaged dataset rather than a single upstream competition's own terms.\n\n## Who publishes it\n\nOmni-MATH was introduced by Bofei Gao, Feifan Song, Zhe Yang and eighteen coauthors across Peking\nUniversity and Alibaba, posted to arXiv in October 2024 and accepted at ICLR 2025. The authors\nmaintain both the GitHub repository and a project page (omni-math.github.io) that hosts the\nleaderboard; HELM and OpenCompass each ship their own scenario/config for the benchmark, pulling\ndirectly from the Hugging Face dataset.\n\n## Lineage\n\nOmni-MATH has no predecessor dataset of its own; it was built as a response to the saturation of\nGSM8K and MATH (`math`, also documented in this batch) rather than as a derived subset of either. No\nsuccessor benchmark to Omni-MATH specifically was identified during this research, though it sits\nalongside other post-2024 olympiad-difficulty math evaluations built for the same reason -- frontier\nmodels outgrowing MATH-500-era benchmarks.\n\n## Saturation and contamination\n\nAt launch (September 2024), OpenAI o1-mini led the official leaderboard at 60.54% overall accuracy,\nfalling to 48.56% restricted to the hardest quarter of problems (difficulty above 5 of 10); o1-preview\nfollowed at 52.55%, with the next tier of open models (Qwen2.5-MATH-72B-Instruct and similar) well\nbehind at 36.20%. The project's own leaderboard has not visibly changed since those September 2024\nentries, so it does not reflect 2025-2026 frontier models, and no later score was found elsewhere\nduring this research -- current saturation status is not established. Contamination risk is medium:\nindividual competition problems circulate publicly and may appear piecemeal in pretraining data, but\nthe specific curated set, with its domain and difficulty labels, has only been packaged and\ndownloadable since September 2024.\n\n## How to run it\n\nHELM's `omni_math` scenario and OpenCompass's `omni_math` configs both load `KbsdJames/Omni-MATH`\ndirectly and implement their own grading rather than calling the authors' GPT-4o-judge or Omni-Judge\npipelines verbatim, so a score reported from either harness is not guaranteed to match the official\nleaderboard's methodology. No lm-evaluation-harness or Inspect Evals task was confirmed for Omni-MATH\nduring this research. Because grading is LLM-judged rather than exact-match, reported scores depend\non which judge model and prompt produced them; always check whether a number came from the official\nGPT-4o-judged leaderboard, the Omni-Judge model, the December 2024 rule-based evaluator, or a\nthird-party harness's own grader before comparing two Omni-MATH scores.\n\n## Reading the numbers\n\nA high Omni-MATH score is decent evidence a model can solve genuinely hard, olympiad-tier problems\nrather than the routine competition mathematics that MATH-500 and GSM8K now saturate on -- at launch,\neven the best reasoning models cleared barely 60%, with accuracy nearly halving on the hardest\nquarter of problems. Because grading depends on an LLM judge, an unfavourably-phrased correct answer\ncan be marked wrong and a plausible-looking wrong one can slip through, so a single score is noisier\nthan a fixed-format multiple-choice one. The benchmark's own leaderboard is roughly two years stale\nrelative to the frontier, so treat any current-model number as coming from the reporting lab's own\nrun rather than a verified leaderboard entry, and check which grader produced it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "omni_math",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "OmniMATH",
   "Omni-Math"
  ],
  "category": "math",
  "contamination": {
   "note": "Problems are drawn from public competition archives and the AoPS community wiki, some of which likely appear individually in pretraining corpora, but the specific 4,428-problem selection, domain labels and reference solutions have only been packaged together and downloadable since September 2024, limiting exact-set memorization for models trained before that date.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 4428,
   "size_note": "4,428 problems in a single public \"test\" split (confirmed via the Hugging Face datasets-server), each with a domain label, a difficulty rating, the problem statement, a full solution and a final answer.\n",
   "splits": "single 'test' split, 4,428 rows; no separate train split is published",
   "url": "https://huggingface.co/datasets/KbsdJames/Omni-MATH"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "omni_math",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "omni_math",
   "other": "Reference grading uses either GPT-4o as an LLM judge (the method behind the official leaderboard) or \"Omni-Judge\", a Llama-3-8B-Instruct model the authors fine-tuned on GPT-4o's judgments (about 95% agreement with GPT-4o on a held-out internal test set); a separate rule-based evaluator covering a subset of the problems was released in December 2024.\n"
  },
  "id": "omni_math",
  "last_updated": "2024-12",
  "leaderboard_url": "https://omni-math.github.io/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Omni-MATH gives a model an olympiad-level mathematics competition problem -- drawn from international, national and regional competitions such as the IMO, Putnam, USAMO and HMMT -- and asks for a full solution and final answer, with no answer choices. Problems are categorised into 33-plus sub-domains (algebra, number theory, geometry, combinatorics, calculus and more) and assigned one of ten difficulty levels, calibrated mainly against the Art of Problem Solving community's own difficulty ratings. It targets reasoning clearly beyond grade-school or standard competition mathematics: the authors built it specifically because GSM8K and the original MATH dataset were, by their account, already being solved with high accuracy by 2024-era models.\n",
  "metric": {
   "baseline_note": "No random-guess or human baseline is established; free-response olympiad problems have no meaningful chance rate and the paper reports no controlled human trial. At launch (September 2024), the strongest reported model, OpenAI o1-mini, reached 60.54% overall accuracy, and 48.56% restricted to problems above difficulty level 5 of 10.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (LLM-judged answer equivalence)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Omni-MATH",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2410.07985",
   "title": "Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models",
   "url": "https://arxiv.org/abs/2410.07985",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Bofei Gao",
    "Feifan Song",
    "Zhe Yang",
    "Zefan Cai",
    "Yibo Miao",
    "Qingxiu Dong",
    "Lei Li",
    "Chenghao Ma",
    "Liang Chen",
    "Runxin Xu",
    "Zhengyang Tang",
    "Benyou Wang",
    "Daoguang Zan",
    "Shanghaoran Quan",
    "Ge Zhang",
    "Lei Sha",
    "Yichang Zhang",
    "Xuancheng Ren",
    "Tianyu Liu",
    "Baobao Chang"
   ],
   "org": "Peking University; Alibaba",
   "url": "https://omni-math.github.io/"
  },
  "released": "2024-09",
  "repo_url": "https://github.com/KbsdJames/Omni-MATH",
  "saturation": {
   "as_of": "2024-09",
   "note": "The project's own leaderboard (omni-math.github.io) has not visibly changed since its September 2024 rows; OpenAI o1-mini leads at 60.54% overall (62.2% under the later rule-based evaluation), with o1-preview second at 52.55%, and the next-best model (Qwen2.5-MATH-72B-Instruct) well behind at 36.20%. No score for a 2025-2026 frontier model (GPT-5, Claude 4.x, Gemini 2.5+) was found on this or any other source during this research, so current standing is not established -- the leaderboard being roughly two years stale is itself worth flagging to anyone citing it as current.\n",
   "status": "open",
   "top_score": 60.54
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models (Gao et al., 2024)",
    "url": "https://arxiv.org/abs/2410.07985"
   },
   {
    "accessed": "2026-09-08",
    "title": "Omni-MATH paper, full text (ar5iv HTML)",
    "url": "https://ar5iv.labs.arxiv.org/html/2410.07985"
   },
   {
    "accessed": "2026-09-08",
    "title": "KbsdJames/Omni-MATH GitHub repository (official, README documents grading methods and news)",
    "url": "https://github.com/KbsdJames/Omni-MATH"
   },
   {
    "accessed": "2026-09-08",
    "title": "KbsdJames/Omni-MATH dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/KbsdJames/Omni-MATH"
   },
   {
    "accessed": "2026-09-08",
    "title": "Omni-MATH project page and leaderboard",
    "url": "https://omni-math.github.io/"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM omni_math_scenario.py",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/omni_math_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass omni_math dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/omni_math"
   }
  ],
  "status": "active",
  "subcategory": "olympiad mathematics",
  "summary": "A 4,428-problem olympiad-level mathematics benchmark built after GSM8K and MATH became easy for frontier models, graded by an LLM judge rather than exact string match.",
  "tags": [
   "math",
   "olympiad",
   "llm-judge",
   "free-response",
   "chain-of-thought"
  ],
  "task_format": "Free-response: read an olympiad-level mathematics problem, produce a full solution and final answer; graded by comparing the extracted final answer to a reference answer."
 }
}