SuperGLUE MultiRC: decide which candidate answers to a question are true given a paragraph that requires more than one sentence.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | multi-sentence reading comprehension with multiple true/false answers (SuperGLUE) |
| Page status | active |
| Metric | official SuperGLUE: F1 over answer-options (F1a) and exact match of each question's gold set (EM); harnesses often report answer-level accuracy instead |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 4848 |
| Dataset licence | other |
| Publisher | University of Pennsylvania / UIUC Cognitive Computation Group (original MultiRC); SuperGLUE from New York University and collaborators |
SuperGLUE MultiRC is a reading-comprehension task. Each item is a paragraph, a question about that paragraph, and several candidate answers. Any number of those answers can be true. The model must label each candidate true or false. Questions are written so that one sentence in the paragraph is not enough. SuperGLUE adopted MultiRC (Khashabi et al., NAACL 2018) because of that multi-sentence design and because the true/false API matches other SuperGLUE tasks better than span extraction. Paragraphs come from seven domains, including news, fiction, and historical text. The language is English.
Per candidate answer, true/false given paragraph and question; English; multiple gold answers per question allowed.
No model card in ModelSpec reports this benchmark yet.