BIG-bench task: given an English short answer and its paragraph, choose which paraphrased question that answer actually satisfies.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench multiple-choice: pick the paraphrased SQuAD question that matches a short answer in context |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1590 |
| Dataset licence | Apache-2.0 |
| Publisher | BIG-bench collaboration (Google); task author Kaustubh Dhole |
question_selection gives a short English answer and the paragraph it came from, then asks which of several paraphrased questions that answer belongs to. Gold questions are paraphrases of SQuAD evaluation items. Negatives are other questions from the same paragraph that share question type and do not overlap in answer span. The skill is matching an answer to an interrogative, not span extraction. It is not [squad](squad.md) and is not in [bbh](bbh.md).
JSON multiple choice. Preferred metric multiple_choice_grade. task_prefix asks the model to pick the question whose answer is the given short answer. Choice count varies (2–10). Canary GUID embedded. Dummy-model header: 1582 multiple-choice queries.
No model card in ModelSpec reports this benchmark yet.