A programmatic BIG-bench generator of nested integer sums, differences and products; BBH freezes 250 two-level expressions as Multistep Arithmetic Two.
unassessed
| Category | math |
|---|---|
| Subcategory | BIG-bench (and BBH) nested integer arithmetic with + − × |
| Page status | active |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1200 |
| Dataset licence | Apache-2.0 (BIG-bench task.py); MIT (BIG-Bench Hard freeze, per that repository) |
| Publisher | Universidad Complutense de Madrid (BIG-bench collaboration) |
multistep_arithmetic asks for the integer value of a fully parenthesised expression built from single-digit integers and the operators +, − and ×. Depth of nesting and the number of operands inside the innermost parentheses vary. Author P. A. M. Casares generated items on the fly in task.py so the original BIG-bench run is a zero-shot exact-string match against Python eval. The task is sequential arithmetic, not word problems. It is not [arithmetic](arithmetic.md) (GPT-3 digit addition files) and not the [simple_arithmetic_json](simple_arithmetic_json.md) templates.
Free-text numeric completion. BIG-bench default: 100 trials for each of four depth lists ([], [2], [2,2], [2,2,2]) and lengths 2, 3, 4, scored with exact_str_match and output regex [-+]?\d+. Zero-shot. BBH freezes 250 expressions as multistep_arithmetic_two.json. Canary GUID in task.py.
No model card in ModelSpec reports this benchmark yet.