A 149-item BIG-bench table-QA task about named penguins; BIG-bench Hard keeps 146 of the same items.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | table question answering over a small penguin attribute table (BIG-bench and BBH) |
| Page status | active |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 149 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration); task author at Naver Labs |
penguins_in_a_table shows a CSV-style table of penguins (name, age, height, weight), sometimes a second table or a newly added row, and asks a short English question such as who is tallest or how many are younger than eight. Author Jos Rozen (Naver Labs) wrote the items by hand so the model must parse the table, any modifier, and the question together. The README notes that in-table glosses are explanations, not few-shot input-output pairs. BIG-bench Hard keeps 146 of 149 examples because that was the full set, not a 250-cap sample.
Dual scoring in task.json: preferred_score exact_str_match (output_regex first word-like token, stop_string newline) and multiple_choice_grade over four or five names or numbers (append_choices_to_input false). Dummy header: 149 multiple-choice and 149 free-text queries. Canary GUID embedded. BBH adds lettered Options (A)–(E) and, in lm-eval, chain-of-thought few-shot.
No model card in ModelSpec reports this benchmark yet.