A BIG-bench task that infers one of 250 functions on lists of natural numbers from a few input/output pairs and applies it to a new list.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench program induction over 250 list-of-integers functions |
| Page status | unknown |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Dataset size | 8000 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
list_functions is a program-induction probe, not a coding benchmark that asks for source. Each subtask names one function over lists of natural numbers (identity, drop ends, unique elements, cumulative sum, and 246 others). The model sees several input/output list pairs, then must emit the output list for a final input. Joshua S. Rule, Steven T. Piantadosi, and Joshua B. Tenenbaum built 250 hand-picked functions from Rule's 2020 MIT thesis so that human learners and symbolic program-induction models already have published per-function accuracies.
Free-text generation of a bracketed integer list. Preferred metric exact_str_match; also bleu and rouge. Parent task.json holds name and keywords; items live in c001–c250, 32 pairs each. Canary GUID embedded. Designed for zero-, one-, or many-shot use of those 32 pairs.
No model card in ModelSpec reports this benchmark yet.