{
 "body": "\n## What it measures\n\nSVAMP is a stress test for elementary English math word problems. Each item is a short story plus a question about one unknown quantity, using only arithmetic. The authors showed that solvers of MAWPS and ASDiv-A could often ignore the question, or ignore word order, and still look strong. SVAMP applies controlled edits so that those shortcuts fail.\n\nThe three edit types are question sensitivity, reasoning ability, and structural invariance. A typical change swaps who received the objects, or rewrites the story without changing the math. The skill under test is reading the actual question, not recalling a template from ASDiv-A.\n\n## How it is scored\n\nThe original paper scores equation or value accuracy of classic seq2seq and tree decoders. Current LLM harnesses score the final number. OpenCompass concatenates body and question, prepends four worked examples, and runs `Gsm8kEvaluator` after `gsm8k_postprocess`. Gold answers are cast to `int`, so a non-integer gold would not match that loader.\n\nThere is no published human or chance baseline. A 2021 Graph2Tree number is not comparable to a 2026 chain-of-thought LLM run.\n\n## Dataset and licence\n\nThe official file `SVAMP.json` holds 1,000 items (`ID`, `Body`, `Question`, `Equation`, `Answer`, `Type`). The README treats that file as the full challenge set. `ChilleD/SVAMP` on Hugging Face is a later mirror that splits the same 1,000 rows into 700 train and 300 test. Both the GitHub repo and the Hub card state MIT. Answers are public.\n\n## Who publishes it\n\nArkil Patel, Satwik Bhattamishra and Navin Goyal. arXiv 2103.07191 appeared 12 March 2021 (v2 15 April 2021) and was published at NAACL 2021. The code and data remain at arkilpatel/SVAMP. No live official leaderboard was found.\n\n## Lineage\n\nSVAMP is not a split of [GSM8K](gsm8k.md). It is a robustness overlay on ASDiv-A and MAWPS, which have no pages here. Later grade-school sets such as GSM8K ask longer multi-step problems; they do not replace SVAMP's variation tests. [TabMWP](tabmwp.md) adds tables rather than these template edits.\n\n## Saturation and contamination\n\nNo current top score was sourced. The set is five years old, fully public, and made of short arithmetic stories, so contamination risk is high. Treat a near-perfect modern score as expected unless the paper used a private variant.\n\n## How to run it\n\nOpenCompass `--datasets svamp_gen` (abbr `svamp`). That config is four-shot in the prompt despite `ZeroRetriever`. The reference data file is GitHub `SVAMP.json`. Do not compare an OpenCompass GSM8K-style number with the 2021 tree-decoder tables, and do not assume the ChilleD 300-item test split is what a paper used.\n\n## Reading the numbers\n\nSVAMP asks whether the model still solves the problem after a small rewrite. It does not measure algebra, contest math, or table reasoning. Pair it with [GSM8K](gsm8k.md) or [MATH](math.md) if you need harder math. If a card reports SVAMP without saying which 1,000-item file or which 300-item slice, the numbers are not comparable.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "svamp",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Simple Variations on Arithmetic Math word Problems"
  ],
  "category": "math",
  "contamination": {
   "note": "The 1,000 items and answers have been public on GitHub since March 2021 and mirrored on Hugging Face. They are short template-like stories, so memorisation and leakage into pretraining are plausible.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1000,
   "size_note": "The official `SVAMP.json` in arkilpatel/SVAMP contains 1,000 challenge items and is the evaluation set. Hugging Face `ChilleD/SVAMP` is a third-party mirror of the same 1,000 rows split 700/300 train/test. OpenCompass's `SVAMPDataset` reads a JSONL of Body+Question+Answer with no split filter.\n",
   "splits": "official release is one 1,000-item eval set; ChilleD/SVAMP adds train 700 / test 300",
   "url": "https://github.com/arkilpatel/SVAMP"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-022 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-022"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "svamp",
   "other": "OpenCompass config `svamp_gen` (hash `svamp_gen_fb25e4`) uses abbr `svamp`, path `opencompass/SVAMP`, four in-prompt CoT examples, `Gsm8kEvaluator`, and `gsm8k_postprocess`. No lm-evaluation-harness task directory named svamp was found on the harness main branch listing.\n"
  },
  "id": "svamp",
  "last_updated": "2021-04",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "SVAMP (Simple Variations on Arithmetic Math word Problems) tests whether a solver actually reads a short English word problem, or just matches shallow templates. Each item is a one-unknown arithmetic story at about grade four or below. The authors built it by taking seeds from ASDiv-A and MAWPS and applying three kinds of edit: change the question, change the reasoning, or change the surface structure. A model that ignored the question on the old sets still scored well; SVAMP is meant to stop that shortcut.\n",
  "metric": {
   "baseline_note": "No random or human baseline is published. The NAACL 2021 paper reports that then-SOTA tree and graph decoders drop sharply relative to MAWPS and ASDiv-A. OpenCompass reuses Gsm8kEvaluator after casting each gold answer to int.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (exact-match final number)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "SVAMP",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2103.07191",
   "title": "Are NLP Models really able to Solve Simple Math Word Problems?",
   "url": "https://arxiv.org/abs/2103.07191",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Arkil Patel",
    "Satwik Bhattamishra",
    "Navin Goyal"
   ],
   "org": "",
   "url": "https://github.com/arkilpatel/SVAMP"
  },
  "released": "2021-03",
  "repo_url": "https://github.com/arkilpatel/SVAMP",
  "saturation": {
   "as_of": "",
   "note": "No current public leaderboard for SVAMP was opened. The 2021 paper showed then-SOTA models well below their MAWPS/ASDiv-A scores. Grade-school one-unknown arithmetic has since become easy for frontier models, but that later ease was not read from a sourced SVAMP table here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Are NLP Models really able to Solve Simple Math Word Problems? (arXiv abs)",
    "url": "https://arxiv.org/abs/2103.07191"
   },
   {
    "accessed": "2026-09-08",
    "title": "SVAMP paper HTML on ar5iv",
    "url": "https://ar5iv.labs.arxiv.org/html/2103.07191"
   },
   {
    "accessed": "2026-09-08",
    "title": "arkilpatel/SVAMP repository",
    "url": "https://github.com/arkilpatel/SVAMP"
   },
   {
    "accessed": "2026-09-08",
    "title": "SVAMP README (1,000-item challenge set, MIT badge, NAACL 2021)",
    "url": "https://raw.githubusercontent.com/arkilpatel/SVAMP/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "SVAMP MIT LICENSE",
    "url": "https://raw.githubusercontent.com/arkilpatel/SVAMP/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official SVAMP.json (1,000 items counted)",
    "url": "https://raw.githubusercontent.com/arkilpatel/SVAMP/main/SVAMP.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "ChilleD/SVAMP third-party Hub mirror",
    "url": "https://huggingface.co/datasets/ChilleD/SVAMP"
   },
   {
    "accessed": "2026-09-08",
    "title": "ChilleD/SVAMP split info (700 train / 300 test)",
    "url": "https://datasets-server.huggingface.co/info?dataset=ChilleD/SVAMP"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass svamp_gen_fb25e4.py (abbr svamp, 4-shot CoT)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/SVAMP/svamp_gen_fb25e4.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass SVAMPDataset loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/svamp.py"
   }
  ],
  "status": "active",
  "subcategory": "grade-school arithmetic word-problem robustness",
  "summary": "A 1,000-item challenge set of grade-school arithmetic word problems made by applying question, reasoning and structure variations to existing MWPs.\n",
  "tags": [
   "math",
   "word-problems",
   "arithmetic",
   "robustness",
   "english"
  ],
  "task_format": "Free-response English word problem. The model must output a numeric answer. OpenCompass concatenates the `Body` and `Question` fields, prompts with four hardcoded chain-of-thought examples, and extracts the final number with the GSM8K postprocessor.\n"
 }
}