Ten fixed synthetic arithmetic tasks -- 2 to 5 digit addition and subtraction, 2-digit multiplication, one-digit composite expressions -- introduced as one small evaluation in the GPT-3 paper.
unassessed
| Category | math |
|---|---|
| Subcategory | synthetic few-shot arithmetic, GPT-3-derived task family |
| Page status | unknown |
| Metric | exact-match accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 20000 |
| Publisher | OpenAI |
The "arithmetic" task family tests basic numeric computation through natural-language word problems: a prompt states two (or, for the composite task, three) numbers and an operation in English, such as "What is 48 plus 27?", and the model must produce the exact numeric answer. lm-evaluation-harness implements ten such tasks, mirroring the ten tasks OpenAI introduced as one evaluation among many in the original GPT-3 paper: one-digit composite expressions combining addition and multiplication (e.g. 6+(4*8)), 2-, 3-, 4- and 5-digit addition, 2-, 3-, 4- and 5-digit subtraction, and 2-digit multiplication. The GPT-3 authors framed this as a test of whether a language model could perform simple, unseen computations "on the fly" from its few-shot context, not as a benchmark of the multi-step mathematical reasoning that GSM8K or MATH later targeted.
A natural-language arithmetic word problem (e.g. 'Q: What is 48 plus 27? A:') in; the model must generate the exact numeric answer as free text, with no chain-of-thought or intermediate steps required or scored.
No model card in ModelSpec reports this benchmark yet.