Color

A 4,000-item BIG-bench task that maps RGB, HEX, HSL, and HCL encodings to one of ten English color names.

Also known as: BIG-bench color

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
SubcategoryBIG-bench 10-way color naming from RGB, HEX, HSL, and HCL (4,000 items)
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size4000
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration); task author at Georgia Tech

What it measures

color shows a digital color encoding and asks for the closest high-level English name among ten options (white, gray, black, red, orange, yellow, green, blue, purple, brown). Zijie J. Wang built four subtasks — rgb, hex, hsl, and hcl — to test whether a language model can read web-style color codes, not whether it can reason about colored objects in a scene.

Task format

Ten-way multiple choice with preferred metric multiple_choice_grade; rouge, bleu, and exact_str_match are also listed. append_choices_to_input is true on each subtask JSON. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub