522 Putnam contest problems scored by boxed exact match, plus 100 functional variants meant to catch memorisation of public contest write-ups.
unassessed
| Category | math |
|---|---|
| Subcategory | university Putnam contest problems with boxed answers and functional variants |
| Page status | active |
| Metric | exact_match (boxed answer, SymPy equivalence) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 522 |
| Dataset licence | Apache-2.0 |
| Publisher | Stanford University, Department of Computer Science |
Putnam-AXIOM tests whether a model can solve undergraduate contest mathematics that still separates today's systems. Each item is a William Lowell Putnam problem from 1938–2023 rewritten so the solution ends in one boxed numeric or algebraic answer. The original 522-item set is a static exam. A companion variation protocol rewrites 100 of those problems by changing variables, constants, and surface wording while keeping the same reasoning. The paper reports that even strong models drop when the numbers change, which is the point of the variation split: to show scores that may come from having seen the public contest archive rather than from solving a new instance.
English LaTeX problem statement. The model writes a solution and a final answer inside \\boxed{}. lm-eval uses a 4-shot Minerva-style prompt, greedy generate_until, and a SymPy/LaTeX equivalence check on the boxed string.
No model card in ModelSpec reports this benchmark yet.