Key/Value Maps

101 Yes/No BIG-bench questions on whether formal statements about key/value maps hold.

Also known as: key-value maps

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	BIG-bench Yes/No formal statements about maps from keys to values (101 items)
Page status	unknown
Metric	multiple_choice_grade
Direction	higher_is_better
Unit	%
Dataset size	101
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration); task author at MIT

What it measures

key_value_maps asks whether short English theorems about partial maps (key/value dictionaries) follow from a stated definition of extends, agree-on, and only-differ. The author wrote the English from scratch, with some lemmas inspired by Coq files in mit-plv/coqutil. The skill is precise logical English, not coding the maps. It is not in [bbh](bbh.md).

Task format

Two-option multiple choice (Yes/No), preferred metric multiple_choice_grade, append_choices_to_input false. Four JSON subtasks share a canary GUID. Dummy-model header: 101 multiple-choice queries.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub