The original 12,500-problem competition-mathematics benchmark from Hendrycks et al. 2021; its 5,000-problem test split is the parent of the smaller MATH-500 subset most current model cards actually report.
unassessed
| Category | math |
|---|---|
| Subcategory | competition mathematics |
| Page status | active |
| Metric | accuracy (pass@1, boxed-answer match) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 5000 |
| Dataset licence | MIT |
| Publisher | UC Berkeley |
MATH gives a model a written competition mathematics problem -- spanning algebra, geometry, number theory, counting and probability, precalculus, intermediate algebra and prealgebra, each labelled with a difficulty level from 1 to 5 -- and asks for a full worked solution ending in a final answer. There are no answer choices. It tests the same skill as MATH-500 (this repository's `math_500` page, a fixed 500-problem subset of MATH's test split): multi-step symbolic and numeric reasoning, not speed, tool use, or any language beyond English. What differs between MATH and MATH-500 is size and provenance, not task design -- they test the identical kind of problem at 5,000 versus 500 items.
Free-response: read a competition mathematics problem, produce a worked solution and a final answer, conventionally inside \boxed{}.
No model card in ModelSpec reports this benchmark yet.