Conceptual Combinations

A 103-item BIG-bench Lite four-way task that asks which English sentence uses a two-word conceptual combination correctly.

Also known as: BIG-bench conceptual_combinations

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	BIG-bench Lite four-way conceptual-combination sentences (103 items, six subtasks)
Page status	unknown
Metric	multiple_choice_grade
Direction	higher_is_better
Unit	%
Dataset size	103
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration)

What it measures

conceptual_combinations gives a short English prompt naming a two-word combination (sometimes with invented-word definitions) and four sentences. The model must pick the sentence that uses the combination felicitously. Six JSON subtasks cover emergent properties, invented words, surprising combinations, homonyms, apparent contradictions, and fictional scenarios. Authors: Dimitri Coelho Mollo, Raphaël Millière, Charles Rathkopf, and Catherine Stinson. English text. Not a metaphor-paraphrase task.

Task format

Four-option multiple choice, preferred metric multiple_choice_grade. Parent task.json holds name and keywords only; items live in six subtask JSON files. Canary GUID embedded. BIG-bench Lite member.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub