{
 "body": "\n## What it measures\n\nTabMWP is a grade-level math exam where every problem comes with a table. The model must read the table and the question together, then produce either a number or a multiple-choice span. Tables are short open-domain grids (prices, counts, schedules), not financial reports.\n\nThe paper ships each table three ways: an image, a text rendering, and a structured object. OpenCompass uses the text table. Free-text items need an integer or decimal; multiple-choice items need the correct option. Gold solutions are annotated so a method can be judged on the answer, not on a full proof.\n\n## How it is scored\n\nAccuracy is the fraction of test items whose predicted answer matches the gold after normalisation. The authors also break out free-text versus multiple-choice, and integer versus decimal versus extractive versus boolean. Human raters on Mechanical Turk scored 90.22% on the test split.\n\nOpenCompass's `TabMWPEvaluator` strips currency marks, compares numbers, and matches choice letters when options exist. Its default config is zero-shot (`ZeroRetriever`). The paper's PromptPG method instead learns which two training examples to put in a GPT-3 prompt. Those protocols are not interchangeable.\n\n## Dataset and licence\n\n38,431 problems, split 23,059 / 7,686 / 7,686 (train / development / test). The paper and `splits.json` agree. 74.7% free-text, 25.3% multiple-choice. Data live under `data/tabmwp` in lupantech/PromptPG. The repo `LICENSE.md` is MIT for the code (copyright Pan Lu 2022). The README states the TabMWP dataset itself is CC BY-NC-SA 4.0. Test answers are public.\n\n## Who publishes it\n\nPan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark and Ashwin Kalyan. arXiv 2209.14610 appeared 29 September 2022; the comment on v3 says ICLR 2023. The project page is promptpg.github.io. The GitHub README still hosts the community leaderboard.\n\n## Lineage\n\nTabMWP is not a table overlay of [GSM8K](gsm8k.md) or [SVAMP](svamp.md). Those sets have no tables. The PromptPG paper is the method; TabMWP is the dataset. No successor id exists here. Tool-using later papers (Chameleon, Program-of-Thoughts) reuse the same test split.\n\n## Saturation and contamination\n\nThe official leaderboard already lists GPT-4 tool agents above 98%, past the 90.22% human mark, so the public test split is saturated for that protocol. The full labelled set has been on GitHub since 2022, so contamination risk is high.\n\n## How to run it\n\nDownload `data/tabmwp` from PromptPG. OpenCompass `--datasets TabMWP_gen` reads `./data/tabmwp/` and scores with `TabMWPEvaluator`. The authors' `run_gpt3` scripts are the few-shot GPT-3 path. Say whether you used text tables or images, and whether tools were allowed. Hub copies are explicitly distrusted by the OpenCompass loader comment.\n\n## Reading the numbers\n\nA high TabMWP score means the model can do grade-school arithmetic on a small text table. It does not mean it can audit a spreadsheet or solve contest math. Compare tool-using GPT-4 numbers only with other tool-using runs. For table-free arithmetic see [GSM8K](gsm8k.md) and [SVAMP](svamp.md).\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "tabmwp",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Tabular Math Word Problems",
   "PromptPG"
  ],
  "category": "math",
  "contamination": {
   "note": "Train, development and test answers have been public in the PromptPG repo since 2022. Grade-level tables are easy to memorise. Tool-using agents that hit 98%+ no longer have headroom on this split.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-NC-SA-4.0",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 38431,
   "size_note": "Paper and project page: 38,431 problems, split 6:2:2 into 23,059 train, 7,686 development and 7,686 test. Free-text 28,719 (74.7%); multiple-choice 9,712 (25.3%). GitHub `data/tabmwp/splits.json` lists the same 23,059 / 7,686 / 7,686 ids. OpenCompass comments that Hub mirrors parse badly and reads a local `./data/tabmwp/` copy instead.\n",
   "splits": "train 23,059 / dev 7,686 / test 7,686",
   "url": "https://github.com/lupantech/PromptPG"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-022 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-022"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "TabMWP",
   "other": "OpenCompass wrapper `TabMWP_gen` imports `TabMWP_gen_2aef96`. It uses `TabMWPDataset` with `path='./data/tabmwp/'`, `ZeroRetriever`, and `TabMWPEvaluator`. The authors' PromptPG code is the paper's few-shot GPT-3 reference. No lm-evaluation-harness task directory was found.\n"
  },
  "id": "tabmwp",
  "last_updated": "2023-12",
  "leaderboard_url": "https://github.com/lupantech/PromptPG",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "TabMWP (Tabular Math Word Problems) gives the model a table and a grade-school math question about that table. The table is available as an image, as semi-structured text, and as a structured grid. About three quarters of items need a free-text number; the rest are multiple-choice text spans. Each item also has a gold solution. The skill is reading the table and doing the arithmetic, not solving a table-free word problem.\n",
  "metric": {
   "baseline_note": "The paper and GitHub leaderboard report human performance of 90.22% on the test split (84.61% free-text, 93.32% multiple-choice). Heuristic guess is listed at 15.29%. OpenCompass does not use that human protocol; it scores generated text against the gold answer with dataset-specific normalisation.\n",
   "direction": "higher_is_better",
   "human_baseline": 90.22,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "TabMWP",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2209.14610",
   "title": "Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoning",
   "url": "https://arxiv.org/abs/2209.14610",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Pan Lu",
    "Liang Qiu",
    "Kai-Wei Chang",
    "Ying Nian Wu",
    "Song-Chun Zhu",
    "Tanmay Rajpurohit",
    "Peter Clark",
    "Ashwin Kalyan"
   ],
   "org": "",
   "url": "https://promptpg.github.io"
  },
  "released": "2022-09",
  "repo_url": "https://github.com/lupantech/PromptPG",
  "saturation": {
   "as_of": "2023-04",
   "note": "The PromptPG README leaderboard, still showing this order when read on 2026-09-08, lists Chameleon (GPT-4) at 98.78% average on 19 April 2023, above the 90.22% human figure. That is a tool-using GPT-4 result on text tables, not a current frontier rerun.\n",
   "status": "saturated",
   "top_score": 98.78
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "TabMWP / PromptPG paper (arXiv abs)",
    "url": "https://arxiv.org/abs/2209.14610"
   },
   {
    "accessed": "2026-09-08",
    "title": "TabMWP paper HTML on ar5iv",
    "url": "https://ar5iv.labs.arxiv.org/html/2209.14610"
   },
   {
    "accessed": "2026-09-08",
    "title": "PromptPG project page (38,431 problems)",
    "url": "https://promptpg.github.io/"
   },
   {
    "accessed": "2026-09-08",
    "title": "lupantech/PromptPG repository",
    "url": "https://github.com/lupantech/PromptPG"
   },
   {
    "accessed": "2026-09-08",
    "title": "PromptPG README (leaderboard, 38,431, human 90.22%)",
    "url": "https://raw.githubusercontent.com/lupantech/PromptPG/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "PromptPG MIT LICENSE.md",
    "url": "https://raw.githubusercontent.com/lupantech/PromptPG/main/LICENSE.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "TabMWP splits.json (23059 / 7686 / 7686)",
    "url": "https://raw.githubusercontent.com/lupantech/PromptPG/main/data/tabmwp/splits.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass TabMWP_gen_2aef96.py",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/TabMWP/TabMWP_gen_2aef96.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass TabMWPDataset and TabMWPEvaluator",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/tabmwp.py"
   }
  ],
  "status": "active",
  "subcategory": "grade-level math word problems over tables",
  "summary": "38,431 grade-level math word problems that require reasoning over both a short question and an accompanying table, mixing free-text and multiple-choice answers.\n",
  "tags": [
   "math",
   "tables",
   "word-problems",
   "grade-school",
   "english"
  ],
  "task_format": "English generation over a text table plus question. OpenCompass prompts `Table: {table}` then `Question: {question}` with no in-context examples and scores with `TabMWPEvaluator`, which normalises numeric answers and matches choice letters when options are present.\n"
 }
}