A nine-task English sentence-understanding suite that defined pre-LLM benchmarking from 2018; models exceeded its human baseline within about 14 months, and its own successor SuperGLUE replaced it.
unassessed
| Category | composite |
|---|---|
| Subcategory | multi-task English sentence and sentence-pair understanding suite (acceptability, sentiment, paraphrase, similarity, inference) |
| Page status | superseded |
| Metric | GLUE Score: unweighted average of nine per-task scores (accuracy for five tasks, Matthews correlation for CoLA, mean of accuracy and F1 for MRPC and QQP, mean of Pearson and Spearman correlation for STS-B) |
| Direction | higher_is_better |
| Unit | points |
| Dataset size | 69711 |
| Dataset licence | The Hugging Face dataset card gives licence "other": GLUE reuses nine pre-existing datasets, and the card and paper refer users to each one's original licence rather than publishing one unified GLUE licence. |
| Publisher | New York University |
GLUE bundles nine separately scored English sentence- and sentence-pair-classification tasks behind one composite score, plus a hand-built diagnostic set reported on the side. Single-sentence tasks ask whether a sentence is grammatically acceptable (CoLA) or what sentiment it expresses (SST-2). Similarity and paraphrase tasks ask whether two sentences paraphrase each other (MRPC), how similar they are on a 1-5 scale (STS-B), or whether two Quora questions ask the same thing (QQP). Inference tasks recast entailment, reading comprehension and coreference as two- or three-way sentence-pair classification (MNLI, QNLI, RTE, WNLI). The nine tasks vary enormously in size by design, from a few hundred training pairs (WNLI) to hundreds of thousands (QQP, MNLI), so a model must generalise across data-scarce and data-rich tasks rather than simply having enough examples to fine-tune on any single one.
Nine separately trained-and-scored tasks, almost all single-sentence or sentence-pair classification; STS-B alone is a regression. Each task keeps its own metric (see How it is scored) and the nine scores are averaged, unweighted, into one GLUE Score out of 100. A separate, analysis-only diagnostic set (AX) is reported alongside the score but not folded into it.
No model card in ModelSpec reports this benchmark yet.