SuperGLUE's two-way textual-entailment task, reused from GLUE RTE: decide whether a hypothesis is entailed by a premise, scored by accuracy.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | two-way English textual entailment (premise/hypothesis) |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 277 |
| Dataset licence | Hugging Face card licence is "other". SuperGLUE refers users to the original PASCAL RTE dataset licences rather than stating one suite licence. |
| Publisher | New York University (SuperGLUE packaging); original RTE challenges from the PASCAL RTE organisers |
RTE asks whether a short hypothesis is entailed by a short premise, as two-class English sentence-pair classification (entailment versus not_entailment). SuperGLUE uses the same data and format as GLUE RTE: the PASCAL RTE1, RTE2, RTE3 and RTE5 challenge sets, merged and collapsed to two labels. The SuperGLUE paper kept RTE because, even after transfer learning lifted GLUE RTE from near chance to the mid-80s, a gap to human accuracy remained. The items are news and Wikipedia sentences, not long documents.
Binary sentence-pair classification. Harnesses prompt the pair and score accuracy of entailment versus not-entailment. OpenCompass has both generation (A/B) and perplexity (Yes/No) configs; lm-evaluation-harness uses a True/False multiple-choice prompt on the SuperGLUE validation split.
No model card in ModelSpec reports this benchmark yet.