{
 "body": "\n## What it measures\n\nMathBench tests mathematics proficiency at five difficulty stages that mirror school progression:\narithmetic, primary, middle, high school and college. Every stage above arithmetic is tested twice,\non two different things: \"Application\" (can the model solve a problem at that level) and \"Theory\"\n(does the model know the underlying definitions and concepts at that level, independent of solving\nanything). The authors built this split specifically because they judged prior math benchmarks such\nas GSM8K to give only one undifferentiated difficulty signal, unable to distinguish a model that\nknows a theorem from one that can apply it. Arithmetic and primary-level application problems are\nfree-response word problems in the GSM8K style; every other split is four-option multiple choice. All\nstages except arithmetic appear in both Chinese and English.\n\n## How it is scored\n\nCloze splits (arithmetic, primary application) are graded on the extracted final numeric answer.\nMultiple-choice splits use Circular Evaluation (CE): the same question is re-asked four times with\nits option order rotated each time (CE-4), and the model is credited only if it answers correctly on\nall four rotations -- a deliberately stricter bar than plain accuracy, meant to catch models that\nguess or exploit positional bias rather than actually reasoning to an answer. The authors report\nresults with 8-shot chain-of-thought prompting for cloze splits and zero-shot chain-of-thought for\nmultiple-choice splits; running either split with a different shot count or without reasoning is a\ndifferent protocol from the one their own results table used.\n\n## Dataset and licence\n\nMathBench holds 3,709 problems in total, drawn across the five stages and split three ways by a\ntaxonomy of stage, application-versus-theory, and topic. The project's GitHub repository\n(open-compass/MathBench) is released under an Apache-2.0 licence, with problems and answer keys both\npublic -- necessary for Circular Evaluation, which grades locally rather than against a held-out\nserver. The dataset is text-only and bilingual for every stage but arithmetic, which is English-only.\n\n## Who publishes it\n\nMathBench comes from Hongwei Liu, Zilong Zheng, Yuxuan Qiao, Haodong Duan, Zhiwei Fei, Fengzhe Zhou,\nWenwei Zhang, Songyang Zhang, Dahua Lin and Kai Chen, primarily at Shanghai AI Laboratory (with some\nauthors also affiliated with Beihang University and Nanjing University), posted to arXiv in May 2024\nand accepted to ACL 2024 (Findings). Several of the same authors -- Kai Chen, Songyang Zhang and\nWenwei Zhang -- also appear on this repository's `livemathbench` page, a separate OpenCompass-team\nbenchmark built around contamination resistance rather than hierarchical difficulty. The team\nmaintains both the GitHub repository and a public leaderboard.\n\n## Lineage\n\nMathBench names no formal predecessor but positions itself explicitly against single-difficulty math\nbenchmarks like GSM8K, which it argues cannot separate conceptual knowledge from problem-solving\nskill. It has no successor or variant tracked in this repository. Readers should not confuse\nMathBench (this page, a hierarchical theory-and-application suite) with `mathqa` (a much older,\n2019, multiple-choice math word-problem dataset unrelated to it beyond the shared word \"math\") or\n`matbench` (a materials-science property-prediction benchmark with no connection to mathematics\neducation at all) -- the three names are easy to mistake for one another but test unrelated things.\n\n## Saturation and contamination\n\nAt the paper's own May 2024 release, the top model (GPT-4o-2024-05-13) already scored 87.7%\napplication / 92.2% theory at the primary stage -- close to a ceiling -- while its college-level\nscores were markedly lower (54.0% application / 85.6% theory), and open-source models trailed further\nat college level. That spread suggests the easiest stages were saturating at release while\ncollege-level application kept separating models. No current (2025-2026) score was found in the\nsources opened for this page, so present-day saturation is not established beyond this dated picture.\nContamination risk is high: all problems and answers have been public under a permissive licence\nsince March 2024, ample time to reach later pretraining corpora, and no contamination study specific\nto MathBench was found.\n\n## How to run it\n\nOpenCompass is the reference harness, registering the suite under its `MathBench` config directory\nwith task abbreviations combining stage and format, for example `mathbench-college-single_choice_cn`\nor `mathbench-primary-cloze_en`. The config supports both a chain-of-thought and a direct-answer\nprompt for multiple-choice splits, and both zero-shot and few-shot settings; the authors' own results\ntable used zero-shot CoT for multiple choice and 8-shot CoT for cloze splits specifically, so scores\nrun under other settings are not directly comparable to that table. No lm-evaluation-harness, HELM,\ninspect_evals or BIG-bench registration was confirmed during this research.\n\n## Reading the numbers\n\nAlways read a MathBench score alongside its stage and split: a strong \"MathBench\" number with no\nfurther detail could mean anything from near-ceiling primary-school arithmetic to a genuinely\nseparating college-level application score, and Theory and Application at the same stage measure\ndifferent things that do not necessarily move together. Because multiple-choice splits use Circular\nEvaluation rather than plain accuracy, a MathBench multiple-choice score is not directly comparable to\na same-named-sounding accuracy figure from a benchmark that does not rotate options. Given the fully\npublic dataset and answer key, treat a high score as a proficiency signal rather than strong evidence\nof contamination-free reasoning.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mathbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "math",
  "contamination": {
   "note": "All 3,709 problems and their answers are public in the GitHub repository under an Apache-2.0 licence, needed there because Circular Evaluation grades locally against the shipped answer key rather than a held-out server. The benchmark has been public since March 2024, giving ample time for inclusion in later pretraining corpora; no contamination study specific to MathBench was found in the sources opened for this page.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en",
    "zh"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 3709,
   "size_note": "3,709 problems total (per the maintainers' own repository), each labelled with a three-level taxonomy (stage, application-or-theory, and topic). Application and Theory questions are counted together in this total; the repository does not publish a separate per-stage or per-split count table in the sources opened for this page.\n",
   "splits": "5 stages (arithmetic, primary, middle, high, college) x up to 2 splits (application, theory) x up to 2 languages (en, zh); arithmetic is English-only and has no theory split",
   "url": "https://github.com/open-compass/MathBench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "MathBench (directory name); tasks are abbreviated mathbench-<stage>-<format>, for example mathbench-college-single_choice_cn or mathbench-primary-cloze_en, combining the 5 stages with the single_choice_cn/en and cloze_cn/en format variants",
   "other": "The maintainers' own GitHub repository (open-compass/MathBench) is the reference source for the problem set and the published results table. OpenCompass's config supports both zero-shot and few-shot (8-shot) prompting and both a with-reasoning and a direct-answer prompt variant for multiple-choice splits; the authors' own reported results used zero-shot CoT for multiple choice and 8-shot CoT for cloze splits, so a score run under different shot or reasoning settings is not directly comparable to the paper's own table.\n"
  },
  "id": "mathbench",
  "last_updated": "2024-05",
  "leaderboard_url": "https://open-compass.github.io/MathBench/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MathBench tests mathematics proficiency at five difficulty stages that mirror school progression: arithmetic, primary, middle, high school and college. Each stage (other than arithmetic) is tested twice, on two different things: \"Application\" problems (can the model solve a problem at that level) and \"Theory\" questions (does the model know the underlying concepts and definitions at that level, independent of solving anything). The authors built this specifically because prior math benchmarks like GSM8K, in their view, gave only a single undifferentiated difficulty signal. Arithmetic and primary-level application problems are free-response word problems; every other stage and the theory questions throughout are four-option multiple choice. All stages except arithmetic are presented in both Chinese and English.\n",
  "metric": {
   "baseline_note": "CE is designed to be harder to pass by guessing than plain accuracy, since a model must answer the same underlying question correctly across all 4 option orderings; the authors do not state a single random-baseline percentage for CE because it depends on option count and is not a simple 1/n calculation. No human baseline was found in the sources opened for this page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy under Circular Evaluation (CE) for multiple-choice splits; plain accuracy for cloze splits",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "MathBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2405.12209",
   "title": "MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark",
   "url": "https://arxiv.org/abs/2405.12209",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Hongwei Liu",
    "Zilong Zheng",
    "Yuxuan Qiao",
    "Haodong Duan",
    "Zhiwei Fei",
    "Fengzhe Zhou",
    "Wenwei Zhang",
    "Songyang Zhang",
    "Dahua Lin",
    "Kai Chen"
   ],
   "org": "Shanghai AI Laboratory, with contributing authors at Beihang University and Nanjing University",
   "url": "https://github.com/open-compass/MathBench"
  },
  "released": "2024-03",
  "repo_url": "https://github.com/open-compass/MathBench",
  "saturation": {
   "as_of": "2024-05",
   "note": "At the paper's own May 2024 release, the top model (GPT-4o-2024-05-13) already scored 87.7% application / 92.2% theory at the primary stage, close to a ceiling, while college-level scores for the same model were far lower (54.0% application / 85.6% theory) and open-source models trailed further (e.g. Qwen2-72B-Instruct 46.3% college application). That spread suggests the easy stages were already saturating at release while college-level application still separated models. No current (2025-2026) score was found in sources opened for this page, so present-day saturation is not established here beyond this dated, stage-by-stage picture.\n",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmark",
    "url": "https://arxiv.org/abs/2405.12209"
   },
   {
    "accessed": "2026-09-08",
    "title": "MathBench (full text, ar5iv) -- author affiliations",
    "url": "https://ar5iv.labs.arxiv.org/html/2405.12209"
   },
   {
    "accessed": "2026-09-08",
    "title": "open-compass/MathBench repository (README, LICENSE, results table)",
    "url": "https://github.com/open-compass/MathBench"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MathBench config (stages, formats, Circular Evaluation wiring)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/MathBench/mathbench_2024_gen_19e486.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MathBench prompt templates (bilingual, with/without reasoning)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/MathBench/mathbench_prompt.py"
   }
  ],
  "status": "active",
  "subcategory": "hierarchical, bilingual theory-and-application mathematics evaluation",
  "summary": "A bilingual, 3,709-problem suite spanning five education stages from arithmetic to college, each scored separately on theory recall and applied problem-solving using circular multiple-choice evaluation.",
  "tags": [
   "math",
   "bilingual",
   "hierarchical",
   "circular-evaluation",
   "multiple-choice",
   "theory-vs-application"
  ],
  "task_format": "Two formats depending on stage and split: free-response cloze problems (arithmetic, primary application) graded on the final extracted number, and four-option multiple-choice questions (middle/high/college application, and theory questions at every stage) graded with Circular Evaluation -- the same question is re-asked with its option order rotated across 4 rounds (CE-4), and a model is only scored correct on that question if all 4 rotations are answered correctly.\n"
 }
}