Reasoning about Colored Objects (BIG-bench)

2,000 generated BIG-bench questions about colors, left/right order, and counts of household objects; BIG-bench Hard keeps 250.

Also known as: BBH reasoning_about_colored_objects

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	BIG-bench (and BBH) referring-expression questions over listed colored objects
Page status	unknown
Metric	multiple_choice_grade
Direction	higher_is_better
Unit	%
Dataset size	2000
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration)

What it measures

reasoning_about_colored_objects lists colored household objects on a desk, table, nightstand, or floor, then asks a short English question. Authors Daphne Ippolito and Chris Callison-Burch generated the items. Skills are referring expressions, left/right order, and counts after removals, not pixel vision. Distinct from BIG-bench [color](color.md), which names RGB/HEX codes. BIG-bench Hard keeps a 250-item sample.

Task format

Multiple choice with preferred metric multiple_choice_grade. append_choices_to_input is false. Choice-set size varies by answer type (yes/no, colors, or numbers). Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub