{
 "body": "\n## What it measures\n\nAIME 2025 repurposes the 30 problems from the 2025 American Invitational Mathematics Examination as a language-model reasoning test. The AIME sits above the AMC 10/12 in the US math-competition pipeline: only students who clear a qualifying score on the AMC are invited to sit it, so the problems assume real fluency in algebra, geometry, number theory and combinatorics rather than rote recall. Each problem has exactly one correct integer answer between 0 and 999, so there is no answer key to pattern-match against and no partial credit for a nearly-right derivation.\n\nBecause the set is small and each problem demands several correct reasoning steps in a row, AIME 2025 rewards models that sustain a chain of algebraic or combinatorial reasoning without a single slip, rather than models that are merely broad.\n\n## How it is scored\n\nScoring is accuracy: the fraction of the 30 problems for which the model's final integer matches the official answer, commonly reported as pass@1 for a single sampled attempt. Some reporters run multiple samples per problem and take an average or majority vote (cons@k) instead of a single pass@1 draw, which can move a score by several points; check which protocol a given report used before comparing numbers directly. Because answers are free-response integers rather than multiple-choice letters, the metric is close to unguessable and there is no meaningfully citable random baseline.\n\nResults also vary with tool access: several trackers report a large jump for the same model when it can write and run Python to check arithmetic, so tool access is part of the evaluation condition, not a footnote.\n\n## Dataset and licence\n\nThe 30 problems split evenly between AIME I 2025 and AIME II 2025, both administered in February 2025. The exam itself is written and owned by the Mathematical Association of America; the MAA does not publish an official machine-readable eval package, so several groups have independently transcribed the problems into eval-ready datasets. The `math-ai/aime25` dataset used by lm-evaluation-harness lists an Apache-2.0 licence for its transcription; a separate mirror, `MathArena/aime_2025`, lists CC BY-NC-SA 4.0 for its own LaTeX-converted, verified version of the same problems. The two disagree because each licences its own transcription work, not the underlying exam, so treat the licence as covering the packaging rather than the problems themselves. All 30 answers are public since the MAA publishes the answer key after each sitting.\n\n## Who publishes it\n\nThe exam is written and administered by the Mathematical Association of America. As a language-model benchmark, no single lab or academic paper owns AIME 2025; it is packaged and tracked independently by several groups, including math-ai (the dataset lm-evaluation-harness pulls from), MathArena (which runs a live leaderboard at matharena.ai), and OpenCompass (`AIME2025`). Model developers also frequently run and report it themselves in system cards and technical reports.\n\n## Lineage\n\nAIME 2025 is one sitting in an annual series; no AIME 2024 page exists yet in this repository, and an AIME 2026 sitting will exist once that exam is held. It is not part of a family page in this repository. Two other competition-math evaluations tracked here, `math_500` and `usamo_2026`, cover different competitions (a curated MATH subset and the harder USA Mathematical Olympiad) rather than being direct successors of AIME 2025.\n\n## Saturation and contamination\n\nAIME 2025 is not yet saturated but is closing in on a ceiling for frontier reasoning models: one tracker recorded GPT-5 at roughly 94.6% closed-book in August 2025, with a tool-augmented variant reaching close to 100%, leaving only two or three of the thirty problems genuinely unsolved by the strongest systems. Because the MAA and community sites publish full problems and solutions immediately after each sitting, the answer key has been public since February 2025; any model trained on a broad web crawl after that date has plausibly seen these exact problems, which is why several trackers already treat the set as an entry-level check rather than a frontier signal. This is also why fresh competition sets such as USAMO 2026 exist alongside it.\n\n## How to run it\n\nlm-evaluation-harness exposes it as the `aime25` task, pulling from the `math-ai/aime25` dataset; OpenCompass exposes an `AIME2025` dataset config. Both are free-response tasks graded by extracting the model's final boxed or stated integer and comparing it to the answer key, so scores are sensitive to prompt format (e.g., whether the model is told to box its final answer) and to whether the harness allows a code-execution tool. Sample count also matters: single-sample pass@1 runs are noisier than reports that average several samples per problem, given there are only 30 problems in total.\n\n## Reading the numbers\n\nA high AIME 2025 score is decent evidence a model can hold together several steps of competition-level algebra or combinatorics without an arithmetic slip, but a 30-problem set has wide sampling noise, so a two- or three-point gap may not be meaningful. Because the set is now public and over a year old, a very high score alone should not be read as proof of general math ability without corroboration from a fresher or held-out set. Check whether a reported score used tools, multiple samples, or a single pass@1 draw before comparing two models' numbers.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "aime_2025",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GLM 5.1",
   "model_id": "zhipu/glm-5-1",
   "provider": "zhipu",
   "provider_display": "Z.ai (Zhipu AI)",
   "score": 95.3,
   "source": "zai-org-model-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-pro",
   "model_id": "openai/o3-pro",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3",
   "model_id": "openai/o3",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-deep-research",
   "model_id": "openai/o3-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini",
   "model_id": "openai/o4-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o4-mini-deep-research",
   "model_id": "openai/o4-mini-deep-research",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "o3-mini",
   "model_id": "openai/o3-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 83.3,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 82.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 79.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 79.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5",
   "model_id": "anthropic/claude-opus-4-5-20251101",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5 (latest)",
   "model_id": "anthropic/claude-opus-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash",
   "model_id": "google/gemini-2-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image",
   "model_id": "google/gemini-2-5-flash-image",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Image (Preview)",
   "model_id": "google/gemini-2-5-flash-image-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite",
   "model_id": "google/gemini-2-5-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 06-17",
   "model_id": "google/gemini-2-5-flash-lite-preview-06-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Lite Preview 09-25",
   "model_id": "google/gemini-2-5-flash-lite-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 04-17",
   "model_id": "google/gemini-2-5-flash-preview-04-17",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 05-20",
   "model_id": "google/gemini-2-5-flash-preview-05-20",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview 09-25",
   "model_id": "google/gemini-2-5-flash-preview-09-2025",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Flash Preview TTS",
   "model_id": "google/gemini-2-5-flash-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 73.3,
   "source": "lmarena.ai, provider-reports"
  }
 ],
 "page": {
  "aliases": [
   "AIME25",
   "AIME 2025 I and II"
  ],
  "category": "math",
  "contamination": {
   "note": "Problems and full worked solutions are published openly by the MAA and mirrored on community wikis within hours of each sitting, so the answer key has been public since February 2025. No source read during this research demonstrated actual memorization by a specific model, so this is scored medium rather than high.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 30,
   "size_note": "30 problems total: 15 from AIME I 2025 and 15 from AIME II 2025",
   "splits": "single test split, 30 rows; no train or validation split",
   "url": "https://huggingface.co/datasets/math-ai/aime25"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "aime25",
   "opencompass": "AIME2025",
   "other": ""
  },
  "id": "aime_2025",
  "last_updated": "",
  "leaderboard_url": "https://matharena.ai/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "AIME 2025 gives a model the problems from the 2025 American Invitational Mathematics Examination, a competition that only the top-scoring AMC participants sit, and checks whether the model returns the single correct integer answer. It exercises multi-step algebra, geometry, number theory and combinatorics reasoning well above the level of grade-school math benchmarks, and gives no partial credit for a sound method that lands on the wrong final number.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (pass@1, exact match on the final integer)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "AIME 2025",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "Mathematical Association of America (MAA)",
   "url": "https://maa.org/maa-invitational-competitions/"
  },
  "released": "2025-02",
  "repo_url": "",
  "saturation": {
   "as_of": "2025-08",
   "note": "GPT-5 scored 94.6% closed-book as of August 2025 per a secondary tracker; the same source reported that adding a Python tool pushed a GPT-5 variant to roughly 100%, and called the set not yet saturated but rapidly approaching a practical ceiling for reasoning models. Treat the exact figure as one tracker's snapshot, not an official leaderboard number.",
   "status": "watch",
   "top_score": 94.6
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "MAA Invitational Competitions",
    "url": "https://maa.org/maa-invitational-competitions/"
   },
   {
    "accessed": "2026-09-07",
    "title": "2025-26 AIME Thresholds Are Now Available - MAA",
    "url": "https://maa.org/news/2025-26-aime-thresholds-are-now-available/"
   },
   {
    "accessed": "2026-09-07",
    "title": "math-ai/aime25 dataset card",
    "url": "https://huggingface.co/datasets/math-ai/aime25"
   },
   {
    "accessed": "2026-09-07",
    "title": "MathArena/aime_2025 dataset card",
    "url": "https://huggingface.co/datasets/MathArena/aime_2025"
   },
   {
    "accessed": "2026-09-07",
    "title": "lm-evaluation-harness AIME task README",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/aime/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "AIME 2025 Benchmark: An Analysis of AI Math Reasoning - IntuitionLabs",
    "url": "https://intuitionlabs.ai/articles/aime-2025-ai-benchmark-explained"
   }
  ],
  "status": "active",
  "subcategory": "competition mathematics",
  "summary": "The 30 problems from the 2025 American Invitational Mathematics Examination, scored as an exact-answer test of competition-level math reasoning.",
  "tags": [
   "math",
   "competition-math",
   "reasoning",
   "exact-match"
  ],
  "task_format": "Free-response competition math problem in, single integer answer from 0 to 999 out; no answer choices are offered."
 }
}