Modified Arithmetic

A BIG-bench task testing whether a model learns arithmetic operations followed by an unusual +1 rule from examples.

Also known as: BIG-bench modified_arithmetic

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorymath
Subcategoryfew-shot arithmetic rule induction
Page statusunknown
Metricexact_str_match
Directionhigher_is_better
Unit%
Dataset size6000
Dataset licenceApache-2.0
PublisherGoogle BIG-bench

What it measures

Modified Arithmetic gives two numbers and five worked examples using an operation. The model must complete a sixth example. In the challenge subtasks the ordinary operation is followed by adding one, while control subtasks omit the extra one.

Task format

Free-text numerical completion across six subtasks: three-digit addition, three-digit subtraction, and two-digit multiplication, each with and without +1.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub