An eight-task English language-understanding suite that replaced GLUE once models exceeded its human baseline, itself since superseded by harder generative and agentic benchmarks.
unassessed
| Category | composite |
|---|---|
| Subcategory | multi-task English sentence and sentence-pair understanding suite, successor to GLUE |
| Page status | superseded |
| Metric | SuperGLUE Score: unweighted average of eight per-task scores (accuracy for five tasks; accuracy and F1 averaged for CB; F1a and exact match averaged for MultiRC; F1 and exact match averaged for ReCoRD) |
| Direction | higher_is_better |
| Unit | points |
| Dataset size | 25730 |
| Dataset licence | The Hugging Face dataset card lists licence "other": SuperGLUE reuses or extends multiple pre-existing datasets (CommitmentBank, COPA, MultiRC, ReCoRD, RTE, WiC, the Winograd Schema Challenge, Winogender) and each retains its own original terms rather than one unified SuperGLUE licence. |
| Publisher | New York University |
SuperGLUE bundles eight separately scored English understanding tasks behind one composite score, chosen specifically to be harder than GLUE's tasks and to cover a wider range of task formats: question answering with yes/no answers over a passage (BoolQ), three-way textual entailment from naturally occurring discourse (CB, CommitmentBank), causal reasoning between two alternatives (COPA), multi-sentence reading comprehension with a variable number of correct answers (MultiRC), span-based reading comprehension requiring cloze-style entity prediction (ReCoRD), two-way textual entailment (RTE), word-sense disambiguation in context (WiC), and coreference resolution via the Winograd schema (WSC). Two further diagnostic sets, AX-b (broad-coverage entailment) and AX-g (Winogender gender-bias probe), are reported alongside the suite but excluded from its main score. All tasks are English text classification or span extraction, not open-ended generation.
Eight separately scored tasks, each single-sentence, sentence-pair, or short-passage classification or span extraction; each keeps its own metric (see How it is scored) and the eight per-task scores are averaged, unweighted, into one SuperGLUE Score out of 100. Two diagnostic sets (AX-b, AX-g) are reported separately and not folded into the score.
No model card in ModelSpec reports this benchmark yet.