A 37k-problem multiple-choice math word problem dataset built by re-annotating AQuA-RAT with interpretable, formal operation programs rather than free-text rationales.
unassessed
| Category | math |
|---|---|
| Subcategory | multiple-choice math word problems with operation-program annotations |
| Page status | active |
| Metric | accuracy and length-normalised accuracy (acc, acc_norm) over the selected option |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 37200 |
| Dataset licence | Apache-2.0 |
| Publisher | University of Washington; Allen Institute for AI |
MathQA gives a model a short math word problem in English plus five lettered answer options and asks it to pick the correct one. The problems themselves come from the earlier AQuA-RAT dataset; MathQA's contribution is a new representation language that annotates each problem with a formal, step-by-step operation program (a sequence of arithmetic operations over the problem's numbers and a small set of constants) rather than AQuA-RAT's original free-text rationale, so that a system's reasoning can be checked structurally rather than only judged by its final answer. As distributed for language-model evaluation, only the question and options are used; the operation-program annotations exist in the data but the standard multiple-choice harness task does not require a model to produce one.
Multiple-choice question answering: a word problem followed by (typically) five lettered options. In lm-evaluation-harness the task is scored by comparing the model's log-likelihood on each option's text as a completion of the question, not by free-text generation.
No model card in ModelSpec reports this benchmark yet.