BIG-bench's text wrapping of Chollet's 400-task ARC evaluation set: infer a hidden grid rule from examples and emit the output grid as digit strings.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | text-serialized grid puzzles from Chollet's ARC, scored as a BIG-bench programmatic task |
| Page status | superseded |
| Metric | exact_str_match (fraction of evaluation puzzles whose every test grid matches exactly); secondary mean_relative_edit_distance |
| Direction | higher_is_better |
| Dataset size | 400 |
| Dataset licence | Apache-2.0 |
| Publisher | BIG-bench collaboration (Google organizers); underlying corpus by François Chollet |
This id is the BIG-bench task that turns François Chollet's Abstraction and Reasoning Corpus into a language-model query. Each evaluation puzzle shows several input/output grids that share a hidden transformation, then a new input grid. The model must emit the matching output grid. BIG-bench writes every grid as a block of digit characters 0-9, one row per line, rather than as coloured cells. The wrapping is meant to test few-shot program-like generalisation on tasks that 2019-era learners could not brute force. It is not the AI2 science-question ARC, and it is not the later ARC Prize / ARC-AGI-2 protocol.
Programmatic BIG-bench task. The prompt is a "TASK QUERY" block of demonstration grids plus a final Input; the model must generate the digit-grid Output. The implementation tries up to three generations per test input and accepts an exact string match. Few-shot settings prepend other training puzzles, not extra demos of the puzzle under test.
No model card in ModelSpec reports this benchmark yet.