{ "body": "\n## What it measures\n\nAIME problems are invitational high-school contest items in algebra, geometry, number theory, and combinatorics. Each problem has one integer answer between 0 and 999. As an LLM benchmark, that format is a strict reasoning test: the derivation can be long, but only the integer counts.\n\nThis id is the family name. It is not a fourth yearly sitting. lm_eval's task `aime` loads a 1983\u20132024 dump. `aime24` / `aime_2024` and `aime25` / `aime_2025` are the 30 problems from those years. `aime_2026` is the 2026 pair of sittings. Do not average a 933-row historical run with a 30-row sitting and call both \"AIME\".\n\n## How it is scored\n\nlm_eval uses greedy generation (`temperature: 0.0`, `max_gen_toks: 32768`) and `exact_match` after pulling a `\\\\boxed{}` or `$...$` span and normalising TeX the way Hendrycks MATH does. Yearly reporters often instead quote pass@1 or cons@k on 30 items, sometimes with a Python tool. Those protocols are not the same number. There is no random baseline.\n\n## Dataset and licence\n\nThe historical pack lm_eval calls `aime` is `gneubig/aime-1983-2024`, compiled from Hemish Veeraboina's Kaggle set and released as CC0-1.0. datasets-server lists 933 train rows; the card README says 2,250 problems. This page records 933 as the loaded size and flags 2,250 as a disagreement. Yearly mirrors (`Maxwell-Jia/AIME_2024`, `math-ai/aime25`) are separate transcriptions, often Apache-2.0 or MIT for the packaging, while the contest itself is an MAA exam. The MAA invitational page returned HTTP 403 in this session, so contest-administration details are taken from the dataset cards rather than from maa.org.\n\n## Who publishes it\n\nThe contest is the Mathematical Association of America's. No single lab owns the LLM eval. Graham Neubig's Hugging Face dump is what `lm_eval --tasks aime` runs. Maxwell Jia and math-ai maintain yearly dumps. MathArena's live board, when opened for this page, listed AIME 2025 and AIME 2026 among final-answer contests and did not list AIME 2024 or the 1983\u20132024 pack.\n\n## Lineage\n\nYearly pages in this repository: `aime_2024`, `aime_2025`, `aime_2026`. `beyondaime` is a different, harder contest-style set, not a sitting of AIME. GSM8K and MATH sit well below this difficulty. USAMO is a proof contest, not this integer format.\n\n## Saturation and contamination\n\nRead saturation on the yearly page, not as one family number. AIME 2024 is already a weak discriminator among reasoning models. The 1983\u20132024 mix is public contest lore and is high contamination risk. A fresh sitting is more informative until its answers have been on the web for a training cycle.\n\n## How to run it\n\n`lm_eval --tasks aime` is the historical pack (all 933 rows as the test set). `aime24` and `aime25` are the yearly YAML files in the same directory. inspect_evals and OpenCompass use underscored yearly names such as `aime_2024`. Always name the year or the pack, the sample count, and whether tools were allowed.\n\n## Reading the numbers\n\nA high score on `aime` (1983\u20132024) mostly says the model can emit AIME-style integers on a large public archive, including many years of leaked solutions. A high score on `aime_2026` is a stronger claim about current contest math, and still not a proof contest. Never compare a 933-item greedy exact-match run with a 30-item cons@64 sitting, and never treat \"AIME\" in a model card as this family unless the card names the year or the pack.\n", "build": { "built_at": "2026-09-09T16:56:50+00:00", "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11", "eligibility_as_of": "2026-09-09" }, "disposition": { "canonical_id": "aime", "reasons": [], "status": "unassessed", "verified_results": [] }, "models_covered": [], "page": { "aliases": [ "American Invitational Mathematics Examination", "AIME 1983-2024" ], "category": "math", "contamination": { "note": "Official answers and worked solutions for past AIME contests are public (contest archives, community wikis, and the machine-readable packs). The 1983\u20132024 mix is the most exposed. Newer sittings start cleaner and then age. BeyondAIME (`beyondaime`) is a different, harder set, not this family.\n", "risk": "high" }, "dataset": { "languages": [ "en" ], "license": "CC0-1.0", "modalities": [ "text" ], "public_test_set": true, "size": 933, "size_note": "Hugging Face datasets-server reports 933 rows in gneubig/aime-1983-2024 (single train split; lm_eval uses that split as the test set). The same card's README claims 2,250 problems; the API size_categories tag is n<1K while cardData says 1K