{
 "body": "\n## What it measures\n\nAIME 2024 repurposes the 30 problems from the 2024 American Invitational Mathematics Examination as a language-model reasoning test. Like every AIME sitting, it sits above the AMC 10/12 in the US math-competition pipeline: only students who clear a qualifying AMC score are invited to sit it, so the problems assume real fluency in algebra, geometry, number theory and combinatorics rather than rote recall. Each problem has exactly one correct integer answer between 0 and 999, so there is no answer key to pattern-match against and no partial credit for a nearly-right derivation.\n\nAIME 2024 was among the first AIME sittings widely adopted as an LLM benchmark, most visibly as the headline math result in OpenAI's September 2024 o1 announcement. That early prominence is also why it is now one of the clearest examples of a competition-math set moving from differentiating to saturated within about a year of reasoning models becoming widespread.\n\n## How it is scored\n\nScoring is accuracy: the fraction of the 30 problems for which the model's final integer matches the official answer, most often reported as pass@1 for a single sampled attempt. Some reporters instead run several samples per problem and take a majority vote (cons@k), which can move a score by a meaningful margin on a set this small; OpenAI's own o1 announcement, for instance, reported 74.4% pass@1 against 83.3% with 64-sample consensus and 93% when re-ranking 1,000 samples for the same model. Because answers are free-response integers rather than multiple-choice letters, there is no meaningfully citable random baseline, and tool access (for example, letting a model run Python to check arithmetic) is part of the evaluation condition rather than a footnote, since it can move scores substantially for the same model.\n\n## Dataset and licence\n\nThe 30 problems split evenly between AIME I 2024, administered January 31-February 1, 2024, and AIME II 2024, administered February 7, 2024, both transcribed with full problems, answer keys and solutions on the Art of Problem Solving wiki shortly after each sitting. As with other AIME years, the MAA does not publish an official machine-readable eval package, so multiple groups have independently transcribed the problems: `math-ai/aime24` carries an Apache-2.0 licence for its transcription, while `Maxwell-Jia/AIME_2024`, the mirror inspect_evals loads, states an MIT licence for its own transcription of the same problems. As with AIME 2025's mirrors, these licences cover each group's packaging work, not the underlying exam, which the MAA itself owns.\n\n## Who publishes it\n\nThe exam is written and administered by the Mathematical Association of America. As a language-model benchmark, no single lab or paper owns AIME 2024; it is packaged and run independently by several groups, including math-ai and Maxwell-Jia (the Hugging Face mirrors most harnesses load), MathArena (which maintains AIME-tracking infrastructure, though AIME 2024 is not among the competitions its live public leaderboard currently surfaces), and OpenCompass (`aime_2024`). Model developers also routinely ran and reported it themselves in system cards and technical reports through 2024 and 2025.\n\n## Lineage\n\nAIME 2024 is the earliest sitting in this repository's AIME series, followed by `aime_2025` and `aime_2026`. Note the id spelling: this repository's existing `aime_2025` page uses an underscore before the year, while this page and `aime_2026` do not; both conventions refer to the same annual AIME series and the inconsistency is a repository data-entry artefact rather than a meaningful distinction, recorded in `benchmarks/_census/DATA-QUALITY.md`. AIME 2024, 2025 and 2026 share an identical format and scoring approach and differ only in which year's 30 problems they use; two other competition-math evaluations tracked in this repository, `math_500` and `usamo_2026`, cover different competitions entirely (a curated MATH subset and the harder, proof-based USA Mathematical Olympiad) rather than being part of this series.\n\n## Saturation and contamination\n\nAIME 2024 is saturated. OpenAI's own September 2024 figures already showed o1 at 74.4-93% depending on sampling strategy, and reasoning-focused models released since then have had over two years to close whatever gap remained; no source read for this page gave a specific current score, but MathArena's decision not to carry AIME 2024 on its live leaderboard (while still tracking AIME 2025 and AIME 2026) and Artificial Analysis's decision to drop the newer AIME 2025 from its Intelligence Index in January 2026 for being saturated both point the same direction. Contamination risk is high: the problems and full worked solutions have been public for well over two years across the MAA, the AoPS wiki and several machine-readable mirrors, ample time to enter web-crawl-derived pretraining data.\n\n## How to run it\n\ninspect_evals exposes it as the `aime_2024` task, loading the `Maxwell-Jia/AIME_2024` dataset and grading by extracting a `\\boxed{}` or final-line answer and comparing it to the key; OpenCompass exposes its own `aime_2024` dataset configuration. Both are free-response tasks, so scores are sensitive to prompt format (for example, whether the model is told to box its final answer), to whether code execution is permitted, and to sample count, given there are only 30 problems in total and a single pass@1 run carries meaningful sampling noise.\n\n## Reading the numbers\n\nBy this page's research date, a high AIME 2024 score mainly confirms that a model is a competent, reasoning-capable system rather than differentiating among frontier models - the set is old and public enough that it functions more as a sanity check than a frontier signal. It remains useful for comparing a new, smaller or non-reasoning model against the well-documented 2024 baselines (GPT-4o's roughly 9-13%, o1's roughly 74-93% depending on sampling), but should not be read as evidence of current state-of-the-art math ability without a fresher, less contaminated set such as `aime_2026` alongside it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "aime_2024",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "AIME24",
   "AIME 2024 I and II"
  ],
  "category": "math",
  "contamination": {
   "note": "AIME 2024's problems and official answer key have been public since February 2024, mirrored across the MAA, the Art of Problem Solving wiki, and multiple machine-readable transcriptions (math-ai/aime24, Maxwell-Jia/AIME_2024, MathArena/aime_2024_I and MathArena/aime_2024_II) for well over two years by this page's research date - long enough that any model trained on a broad web crawl since then has plausibly seen these exact problems and their worked solutions.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 30,
   "size_note": "30 problems total: 15 from AIME I 2024, administered January 31-February 1, 2024, and 15 from AIME II 2024, administered February 7, 2024.",
   "splits": "single test split, 30 rows; no train or validation split",
   "url": "https://huggingface.co/datasets/math-ai/aime24"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "aime_2024",
   "lm_eval": "",
   "opencompass": "aime_2024",
   "other": ""
  },
  "id": "aime_2024",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "aime_2025"
   ],
   "variants": []
  },
  "measures": "AIME 2024 gives a model the problems from the 2024 American Invitational Mathematics Examination, a competition only the top-scoring AMC 10/12 participants are invited to sit, and checks whether the model returns the single correct integer answer. It exercises multi-step algebra, geometry, number theory and combinatorics reasoning well above grade-school math benchmarks, and gives no partial credit for a sound method that lands on the wrong final number. As one of the earliest AIME sittings widely adopted as an LLM benchmark, it is also the sitting most often cited as an example of how quickly a fixed competition-math set can go from differentiating to saturated once reasoning-focused models arrive.",
  "metric": {
   "baseline_note": "OpenAI's own September 2024 announcement of o1 reported GPT-4o scoring 9.3% pass@1 (13.4% with 64-sample consensus) on AIME 2024, while o1 reached 74.4% pass@1 (83.3% consensus, 93% when re-ranking 1,000 samples) - the documented jump that established AIME 2024 as an early reasoning-model benchmark. This is a September 2024 snapshot, not a current figure; see Saturation and contamination.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (pass@1, exact match on the final integer)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "AIME 2024",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "Mathematical Association of America (MAA)",
   "url": "https://maa.org/maa-invitational-competitions/"
  },
  "released": "2024-02",
  "repo_url": "",
  "saturation": {
   "as_of": "2024-09",
   "note": "OpenAI's own September 2024 announcement reported o1 reaching 83.3% (64-sample consensus) and 74.4% (pass@1) on AIME 2024, already a large jump over GPT-4o's 13.4%/9.3%. That is a two-year-old snapshot relative to this page's research date, not a current one. Indirect evidence points to the ceiling having been reached well before now: MathArena's live leaderboard tracks AIME 2025 and AIME 2026 but does not include AIME 2024 among its current competitions, and Artificial Analysis dropped the newer AIME 2025 from its Intelligence Index in a January 2026 refresh specifically for being saturated. No source read for this page gave a specific current top score for AIME 2024 itself, so treat 'saturated' here as a well-evidenced inference rather than a directly-read 2026 figure.",
   "status": "saturated",
   "top_score": 83.3
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MAA Invitational Competitions",
    "url": "https://maa.org/maa-invitational-competitions/"
   },
   {
    "accessed": "2026-09-08",
    "title": "math-ai/aime24 dataset card",
    "url": "https://huggingface.co/datasets/math-ai/aime24"
   },
   {
    "accessed": "2026-09-08",
    "title": "Maxwell-Jia/AIME_2024 dataset card",
    "url": "https://huggingface.co/datasets/Maxwell-Jia/AIME_2024"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals aime_2024 task",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/aime_2024"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass dataset configs (aime_2024 directory confirmed present)",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets"
   },
   {
    "accessed": "2026-09-08",
    "title": "Learning to Reason with LLMs (OpenAI o1 announcement, AIME 2024 results table)",
    "url": "https://openai.com/index/learning-to-reason-with-llms/"
   },
   {
    "accessed": "2026-09-08",
    "title": "2024 AIME I, Art of Problem Solving wiki",
    "url": "https://artofproblemsolving.com/wiki/index.php/2024_AIME_I"
   },
   {
    "accessed": "2026-09-08",
    "title": "2024 AIME II, Art of Problem Solving wiki",
    "url": "https://artofproblemsolving.com/wiki/index.php/2024_AIME_II"
   },
   {
    "accessed": "2026-09-08",
    "title": "MathArena aime competition configs (aime_2024_I.yaml, aime_2024_II.yaml)",
    "url": "https://github.com/eth-sri/matharena/tree/main/configs/competitions/aime"
   }
  ],
  "status": "saturated",
  "subcategory": "competition mathematics",
  "summary": "The 30 problems from the 2024 American Invitational Mathematics Examination, an exact-answer competition-math test now well past its useful ceiling for frontier models.",
  "tags": [
   "math",
   "competition-math",
   "reasoning",
   "exact-match"
  ],
  "task_format": "Free-response competition math problem in, single integer answer from 0 to 999 out; no answer choices are offered."
 }
}