A 38-item BIG-bench task that asks for the part of speech of an ambiguous word in a crash-blossom news headline.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench four-way POS disambiguation of crash-blossom headlines (38 items) |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 38 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
crash_blossom gives one English headline and names a word in it. The model must pick verb, adjective, noun, or preposition for that word. Crash blossoms are headlines whose syntax is funny or preposterous under the wrong parse. The authors rewrote sourced headlines so the strings would not match blog copies. The skill is POS plus world knowledge, not pronoun coreference as in disambiguation_qa.
Four-option multiple choice; preferred metric multiple_choice_grade. task_prefix asks for verb/adjective/noun/preposition. append_choices_to_input is false. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.