A 1,000-item challenge set of grade-school arithmetic word problems made by applying question, reasoning and structure variations to existing MWPs.
unassessed
| Category | math |
|---|---|
| Subcategory | grade-school arithmetic word-problem robustness |
| Page status | active |
| Metric | accuracy (exact-match final number) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1000 |
| Dataset licence | MIT |
SVAMP (Simple Variations on Arithmetic Math word Problems) tests whether a solver actually reads a short English word problem, or just matches shallow templates. Each item is a one-unknown arithmetic story at about grade four or below. The authors built it by taking seeds from ASDiv-A and MAWPS and applying three kinds of edit: change the question, change the reasoning, or change the surface structure. A model that ignored the question on the old sets still scored well; SVAMP is meant to stop that shortcut.
Free-response English word problem. The model must output a numeric answer. OpenCompass concatenates the `Body` and `Question` fields, prompts with four hardcoded chain-of-thought examples, and extracts the final number with the GSM8K postprocessor.
No model card in ModelSpec reports this benchmark yet.