simple_arithmetic_json_multiple_choice (BIG-bench JSON MC arithmetic template)

An 8-item BIG-bench JSON template of one-digit addition with four numeric choices, written as an example of the multiple-choice JSON format.

Also known as: Simple multiple choice arithmetic example using JSON

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorymath
SubcategoryBIG-bench JSON multiple-choice template; one-digit addition
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size8
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench organizers)

What it measures

simple_arithmetic_json_multiple_choice shows a one-digit addition such as "1 + 4 =" and four numeric options, one of which is the true sum. The eight items are the first eight one-digit equations from the free-text JSON template, recast as four-way multiple choice. The organizers state that its primary purpose is to show how a JSON task encodes multiple-choice scoring, not to measure arithmetic. BIG-bench tags it "example task". It is English-script, text-only, and scored zero-shot by ranking the labelled options.

Task format

Multiple-choice addition equation with four numeric options; scored with BIG-bench multiple_choice_grade (likelihood ranking over the options).

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub