The American Invitational Mathematics Examination as an LLM test: exact integer answers to contest problems, packaged as a 1983–2024 historical set and as yearly 30-problem sittings.
unassessed
| Category | math |
|---|---|
| Subcategory | competition mathematics, exact integer answers (historical pack and yearly sittings) |
| Page status | active |
| Metric | exact_match / pass@1 accuracy on the final integer |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 933 |
| Dataset licence | CC0-1.0 |
| Publisher | Mathematical Association of America (exam); LLM packs by Hemish Veeraboina / Graham Neubig (1983–2024), Maxwell Jia (2024), math-ai (2025) |
AIME is a US/Canada invitational math contest. Each sitting has 15 problems and a three-hour limit. Every official answer is an integer from 0 to 999. Language-model evals reuse those problems as free-response reasoning tests: the model must output the integer, with no multiple-choice list and no partial credit. This family page covers that shared format. lm-evaluation-harness task `aime` is specifically the 1983–2024 historical pack (`gneubig/aime-1983-2024`). Tasks `aime24` and `aime25`, and this repository's `aime_2024`, `aime_2025`, and `aime_2026` pages, are single-year 30-problem sittings (AIME I and II).
Free-response contest problem in, single integer 0–999 out. lm_eval extracts a boxed or dollar-delimited answer and scores exact match after light TeX normalisation, greedy decoding, up to 32,768 generated tokens.
No model card in ModelSpec reports this benchmark yet.