{
 "body": "\n## What it measures\n\nUSAMO 2026 evaluates whether a model can produce a complete, rigorous mathematical proof, not just a\nfinal numeric answer, for the six problems of the 2026 USA Mathematical Olympiad. USAMO is a proof-based\ncompetition: every problem asks for a full written argument, so the benchmark tests multi-step\nmathematical reasoning and the ability to communicate a valid proof, in contrast to final-answer-only\nolympiad benchmarks. The domain is competition mathematics and the modality is text, with problems and\nsolutions transcribed in LaTeX.\n\n## How it is scored\n\nEach of the six problems is graded on the competition's own 0-7 point scale, for a maximum of 42 points,\nconverted to a percentage for the leaderboard. The publisher's leaderboard runs each model four times\nper problem. Grading follows a semi-automatic pipeline built on LLM juries, intended to reduce the\nbiases a single LLM judge shows when grading proofs, though the publisher notes that small phrasing\nissues in a model's proof can still mislead a judge and are a recurring source of disagreement. The\npredecessor methodology, used to grade the 2025 USAMO, relied on expert human annotators grading full\nsolutions within hours of release rather than automated judges; a given score's provenance (fully human\nversus semi-automatic) is worth checking, since the two are not guaranteed to agree.\n\n## Dataset and licence\n\nSix problems sourced from the actual 2026 USA Mathematical Olympiad, transcribed to LaTeX and verified,\neach shipped with its point value, a grading rubric and a sample solution. The dataset is published on\nHugging Face as MathArena/usamo_2026 under a CC BY-NC-SA 4.0 licence (attribution, non-commercial,\nshare-alike). Because USAMO problems and solutions become public once the competition runs, this is not\na held-out answer set in the usual sense; its resistance to contamination comes from evaluation timing,\ncovered below, not secrecy.\n\n## Who publishes it\n\nUSAMO 2026 is part of MathArena, run by the SRI Lab at ETH Zurich together with INSAIT, with support\nfrom a Google evaluation grant. The team spans several MathArena papers: the original 2025 USAMO\nevaluation, \"Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad\" (March 2025), by Ivo Petrov,\nJasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunovi\u0107, Nikola\nJovanovi\u0107 and Martin Vechev; the general MathArena platform paper (May 2025) by Balunovi\u0107, Dekoninck,\nPetrov, Jovanovi\u0107 and Vechev; and the current platform paper covering this evaluation, \"Beyond\nBenchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs\" (May 2026), by Dekoninck,\nJovanovi\u0107, Tim Gehrunger, K\u00e1ri R\u00f6gnvaldsson, Petrov, Chenhao Sun and Vechev. The team maintains the live\nleaderboard at matharena.ai.\n\n## Lineage\n\nUSAMO 2026 is the current-year instance of a yearly pattern: MathArena has run a USAMO evaluation for\neach recent olympiad, with the 2025 evaluation as this one's direct conceptual predecessor, using the\nsame grading philosophy each year. It sits inside the broader MathArena platform, which also evaluates\nAIME, CMIMC and IMO among other competitions, and which by 2026 had expanded further to research-level\narXiv problems and formal Lean proof generation. No other MathArena competition ids were found to exist\nin this repository at the time of writing, and this id is specific to the 2026 USAMO, not the series as\na whole.\n\n## Saturation and contamination\n\nThe publisher's own leaderboard, read on the research date, showed the top model at 95.2%, a second\nmodel at 74.4%, then a step down to 47.0% and below, a saturated-at-the-top, open-in-the-middle picture.\nThe May 2026 MathArena paper separately reports an even newer model reaching 98% on this same\nevaluation, reinforcing that leading models are now close to solving USAMO-level proofs. Contamination\nis handled by timing rather than secrecy: MathArena's stated platform-wide approach is to evaluate\nmodels as soon as a competition's problems are released, before they could plausibly enter training\ndata, which is why the publisher describes its competitions as effectively uncontaminated at first\nevaluation. That protection erodes for any model trained after the problems and grading discussion\nbecame public.\n\n## How to run it\n\nEvaluation code and the grading pipeline are maintained in the MathArena team's repository\n(eth-sri/matharena), and the dataset is distributed via Hugging Face as MathArena/usamo_2026. Because\ngrading involves an LLM jury rather than exact-match scoring, two labs re-running the same model can get\nslightly different scores depending on which judge models they use and how many of the four runs per\nproblem they average. This task was not confirmed to be present in lm-evaluation-harness, inspect_evals,\nHELM, OpenCompass or BIG-bench's published task lists at the time of this research.\n\n## Reading the numbers\n\nA high USAMO 2026 score means a model can write a complete, correct mathematical proof under olympiad\nconditions, a meaningfully higher bar than getting the right final number. It does not say how the model\nperforms on problems that don't resemble olympiad style, or on problems published after its training\ncutoff versus before it, and because grading leans on an LLM jury, a score close to another model's may\nreflect judge disagreement rather than a real gap in proof quality. Compare scores from the same grading\npipeline generation and check the as-of date before treating a small difference as meaningful.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "usamo_2026",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 97.6,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GLM 5.1",
   "model_id": "zhipu/glm-5-1",
   "provider": "zhipu",
   "provider_display": "Z.ai (Zhipu AI)",
   "score": 83.8,
   "source": "zai-org-model-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.4,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 42.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  }
 ],
 "page": {
  "aliases": [
   "MathArena USAMO 2026"
  ],
  "category": "math",
  "contamination": {
   "note": "MathArena's stated platform-wide approach is to evaluate models as soon as a competition's problems are released, before they could plausibly enter training data, which the publisher says effectively eliminates contamination at first evaluation. That protection erodes for any model trained after the problems, solutions and grading discussion became public, since USAMO problems are not held out once the competition runs.",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-NC-SA 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 6,
   "size_note": "Six problems from the 2026 USA Mathematical Olympiad, each with a point value, a grading rubric (grading_scheme) and a sample solution.",
   "splits": "single set; a fresh competition instance each year, no train/test split",
   "url": "https://huggingface.co/datasets/MathArena/usamo_2026"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice H",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "usamo_2026",
  "last_updated": "",
  "leaderboard_url": "https://matharena.ai/usamo/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "USAMO 2026 evaluates whether a model can produce a complete, rigorous mathematical proof, not just a final numeric answer, for the six problems of the 2026 USA Mathematical Olympiad. USAMO is a proof-based competition, so each problem asks for a full written argument, testing multi-step mathematical reasoning and the ability to communicate a valid proof rather than final-answer pattern matching.",
  "metric": {
   "baseline_note": "Each of the six problems is graded 0-7 on the competition's own scale, for a maximum of 42 points, converted to a percentage. The publisher's leaderboard runs each model four times per problem.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "% of maximum rubric points (proof grading)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "USAMO 2026",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2605.00674",
   "title": "Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs",
   "url": "https://arxiv.org/abs/2605.00674",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Jasper Dekoninck",
    "Nikola Jovanovi\u0107",
    "Tim Gehrunger",
    "K\u00e1ri R\u00f6gnvaldsson",
    "Ivo Petrov",
    "Chenhao Sun",
    "Martin Vechev"
   ],
   "org": "SRI Lab, ETH Zurich, with INSAIT (MathArena project); evaluation partly supported by a Google grant",
   "url": "https://matharena.ai/"
  },
  "released": "2026-05",
  "repo_url": "https://github.com/eth-sri/matharena",
  "saturation": {
   "as_of": "2026-09",
   "note": "Publisher's live leaderboard (matharena.ai), read on the research date, showed the top model at 95.2%, a second model at 74.4%, then a step down to 47.0% and below: saturated at the very top, still open through the middle of the field. The May 2026 MathArena paper separately reports an even newer model reaching 98% on this same evaluation.",
   "status": "saturated",
   "top_score": 95.2
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Proof or Bluff? Evaluating LLMs on 2025 USA Math Olympiad",
    "url": "https://arxiv.org/abs/2503.21934"
   },
   {
    "accessed": "2026-09-07",
    "title": "MathArena: Evaluating LLMs on Uncontaminated Math Competitions",
    "url": "https://arxiv.org/abs/2505.23281"
   },
   {
    "accessed": "2026-09-07",
    "title": "Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs",
    "url": "https://arxiv.org/abs/2605.00674"
   },
   {
    "accessed": "2026-09-07",
    "title": "MathArena USAMO 2026 leaderboard",
    "url": "https://matharena.ai/usamo/"
   },
   {
    "accessed": "2026-09-07",
    "title": "MathArena/usamo_2026 dataset card",
    "url": "https://huggingface.co/datasets/MathArena/usamo_2026"
   }
  ],
  "status": "active",
  "subcategory": "olympiad proof-writing",
  "summary": "Grades full written proofs, not just final answers, for the six 2026 USA Mathematical Olympiad problems.",
  "tags": [
   "olympiad",
   "proof-grading",
   "contamination-resistant",
   "llm-jury"
  ],
  "task_format": "six open-ended proof problems in LaTeX; a model produces a full written solution, graded against a rubric"
 }
}