A collaborative suite of 200-plus wildly varied tasks from 450 authors; almost every number called BIG-bench today is really BIG-bench Hard or BIG-bench Lite.
unassessed
| Category | composite |
|---|---|
| Subcategory | collaborative multi-task probe of language-model capabilities |
| Page status | superseded |
| Metric | normalized preferred metric (task-specific; averaged across tasks) |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Apache-2.0 |
| Publisher | Google (organizers); a 450-author, 132-institution open collaboration |
BIG-bench does not test one skill. It bundles more than 200 independently authored tasks -- 204 at publication, contributed by 450 authors across 132 institutions -- spanning linguistics, childhood development, mathematics, common-sense reasoning, biology, physics, social bias and software development, among others. The organizers deliberately solicited tasks believed to be beyond the capabilities of language models current at the time, so its coherence comes from a selection criterion (hardness for 2021-era models), not from a shared format or domain.
About 80% of tasks are JSON tasks: fixed lists of input/target examples, scored by a metric the task author chose (exact match, ROUGE, multiple-choice log-probability, and others), which also supports few-shot prompting. The remaining 20% are programmatic tasks written in Python that can query the model over several rounds and score it with custom code, such as the twenty_questions task. Each task author sets a preferred metric plus a high and low score anchor.
No model card in ModelSpec reports this benchmark yet.