GSM8K test problems with one number replaced by a large integer, scored by exact match on the recalculated numeric answer.
unassessed
| Category | math |
|---|---|
| Subcategory | grade-school word problems with large substituted numbers |
| Page status | active |
| Metric | accuracy (exact-match numeric answer) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1319 |
| Dataset licence | MIT |
| Publisher | Language Technologies Institute, Carnegie Mellon University |
GSM-Hard keeps the English grade-school word-problem templates of the GSM8K test set and replaces one number in each question with a random integer of up to seven digits. The intent, from the PAL paper, is to test whether a model that can solve GSM8K is actually doing the arithmetic, rather than exploiting the fact that many GSM8K quantities are small integers. The story and the required operations stay the same; only the magnitude of one operand, and therefore the numeric target, change.
Free-response English word problem; the model must emit a final numeric answer, which reporters extract and score by exact match. PAL additionally scores programs executed in a Python interpreter.
No model card in ModelSpec reports this benchmark yet.