{
 "body": "\n## What it measures\n\nGLUE bundles nine separately scored tasks behind a single composite score: grammatical acceptability\n(CoLA); sentiment (SST-2); whether two sentences paraphrase each other (MRPC) or how similar they are 1-5\n(STS-B); whether two Quora questions are duplicates (QQP); and entailment between a premise and a\nhypothesis, recast from four source tasks into two- or three-way classification (MNLI, QNLI, RTE, WNLI).\nNearly every task is single-sentence or sentence-pair classification in English; only STS-B is a regression.\nA separate, hand-built diagnostic set (AX) tests fine-grained entailment phenomena alongside the main score,\nnot folded into it. The nine tasks deliberately span very different data regimes, from a few hundred\ntraining pairs (WNLI) to hundreds of thousands (QQP, MNLI), to reward models that share knowledge across\ntasks rather than ones that merely have enough data for any single one.\n\n## How it is scored\n\nEach task keeps its own metric: accuracy for SST-2, MNLI, QNLI, RTE and WNLI; Matthews correlation\ncoefficient for CoLA (-1 to 1, 0 the uninformed baseline, chosen for CoLA's unbalanced classes); the mean of\naccuracy and F1 for MRPC and QQP, also class-unbalanced; and the mean of Pearson and Spearman correlation\nfor STS-B. The headline GLUE Score is the unweighted average of those nine numbers, scaled to 0-100, so a\nsingle point does not represent the same underlying quantity from one task to the next.\n\nOfficially, scoring means submitting predictions to the GLUE leaderboard, since eight of the nine tasks'\ntest-set labels were never published (Hugging Face's mirror returns -1 as a dummy label for them); MRPC is\nthe exception, its labels already public before GLUE adopted the corpus. Most papers outside the official\nleaderboard report validation-split scores instead, as lm-evaluation-harness does for every subtask.\n\n## Dataset and licence\n\n69,711 validation rows sum across the nine tasks (MNLI's matched and mismatched dev sets counted\nseparately) -- the split most non-leaderboard numbers actually use. Training splits sum to 949,733 rows,\noverwhelmingly from QQP (363,846) and MNLI (392,702). A separate 1,104-example AX diagnostic set is\ntest-only, hidden-label, and scored outside the nine-task average (see Saturation and contamination). The\nHugging Face dataset card sets licence \"other\": GLUE republishes nine pre-existing datasets and refers users\nto each one's original licence rather than stating one for the whole suite.\n\n## Who publishes it\n\nAlex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy and Samuel R. Bowman, at New York\nUniversity, first presented GLUE at the EMNLP 2018 BlackboxNLP workshop, with a revised version at ICLR\n2019. The benchmark site and leaderboard live at gluebenchmark.com; the authors' own baseline code is\narchived at `nyu-mll/GLUE-baselines` on GitHub.\n\n## Lineage\n\nGLUE has no predecessor -- it repackaged nine pre-existing English NLU datasets under one leaderboard. Its\ndirect successor is SuperGLUE (Wang et al., 2019), built by several of the same authors specifically\nbecause, in that paper's own words, GLUE performance had \"recently surpassed the level of non-expert\nhumans, suggesting limited headroom for further research\"; SuperGLUE does not yet have its own page here.\nGLUE also inspired same-shaped benchmarks this repository catalogues, including CLUE (`clue`, released as\n\"ChineseGLUE\" before its current name) and BasqueGLUE, reused inside `basque_bench`. Three of GLUE's nine\ntasks have subset pages here (`glue_cola`, `glue_mrpc`, `glue_qqp`); the other six and the AX diagnostic do\nnot yet have pages.\n\n## Saturation and contamination\n\nGLUE is saturated, and by its own successor's telling, was saturated almost immediately. The SuperGLUE paper\nrecords GLUE's state of the art as of July 2019 at 88.4 (XLNet), 1.3 points above the 87.1 human-performance\nestimate it cites (Nangia and Bowman, 2019), beating that estimate outright on four of the nine tasks. That\nwas barely 14 months after GLUE's own paper reported GPT and BERT at 72.8 and 80.2 against a 63.7\nno-transfer-learning baseline -- a climb fast enough that GLUE could no longer discriminate between strong\nsystems, exactly the case SuperGLUE's authors made for replacing it. Tellingly, aggregate saturation masked\npersistent weakness elsewhere: even as the composite score passed the human estimate, GLUE's own diagnostic\nset stayed far below its human baseline (0.42 vs 0.80 R3, per SuperGLUE's re-reading of it) -- a saturated\nheadline number coexisting with a real gap on harder linguistic phenomena.\n\nContamination risk is high. Every task's training text, and most tasks' labels, have been continuously\npublic since 2018, built on source corpora mostly already public for years before that, and mirrored\nacross GitHub and Hugging Face since -- exclusion from web-scale pretraining is not something any of the\nnine tasks can claim.\n\n## How to run it\n\nlm-evaluation-harness groups eight of the nine tasks (cola, mnli, mrpc, qnli, qqp, rte, sst, wnli; STS-B\nexcluded) under a `glue` tag, evaluated on the validation split since GLUE's test labels are not public.\nOpenCompass configures each task separately instead -- `GLUE_CoLA`, `GLUE_MRPC`, `GLUE_QQP` and so on -- and\nits CoLA config notably scores plain accuracy rather than Matthews correlation, a real protocol difference\nfrom both the paper and lm-evaluation-harness's own `mcc`-scored `cola` task (see `glue_cola`). Not\nconfirmed in HELM's or BIG-bench's task lists. The GLUE leaderboard (gluebenchmark.com/leaderboard) is\nclient-rendered and did not return scores as static content during this research pass.\n\n## Reading the numbers\n\nA high GLUE Score, on its own, mostly says a model is competent at short-sentence English classification\ncirca 2018-2019 -- not much more. The benchmark was designed to separate systems when 70-80 was a good\nscore; once frontier systems cleared the high 80s it stopped discriminating among them, which is why no\nmodel card in this repository reports it and why SuperGLUE, then harder suites still, took its place. A GLUE\nnumber also hides which of nine unrelated metrics moved it: check the per-task breakdown, since a strong QQP\nF1 and a weak CoLA correlation can average to the same headline figure as the reverse. And treat any GLUE\nnumber as measuring a fixed, dated target rather than open-ended language understanding: the diagnostic set\nshowed models could pass the main score while still failing basic entailment phenomena, so a high GLUE Score\nreads better as \"solved 2018-era sentence classification\" than as \"understands language.\"\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "glue",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "General Language Understanding Evaluation benchmark"
  ],
  "category": "composite",
  "contamination": {
   "note": "Eight of the nine tasks' training data, and all nine tasks' input text if not every label, have been continuously public since GLUE's 2018 release, built on source corpora (the Stanford Sentiment Treebank, the Microsoft Research Paraphrase Corpus, SNLI/MultiNLI, SQuAD, four RTE challenge datasets, Quora Question Pairs) that were mostly already public for years before that. Wide mirroring across GitHub and Hugging Face since 2018 makes exclusion from web-scale pretraining corpora hard to guarantee for any of the nine tasks.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "The Hugging Face dataset card gives licence \"other\": GLUE reuses nine pre-existing datasets, and the card and paper refer users to each one's original licence rather than publishing one unified GLUE licence.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 69711,
   "size_note": "69,711 validation-split rows sum across the nine tasks (MNLI's matched and mismatched dev sets counted separately: 9,815 and 9,832) -- the split most published, non-leaderboard numbers actually use, since GLUE's own test-set labels are undisclosed for eight of the nine tasks (returned as -1 placeholders by the Hugging Face mirror) and are scored only by submitting predictions to the official leaderboard. MRPC is the exception: its Hugging Face test split carries real, publicly disclosed labels, because the corpus already had a public test set before GLUE adopted it. Training splits sum to 949,733 rows, dominated by QQP (363,846) and MNLI (392,702); the smallest task, WNLI, has only 635 training and 71 validation rows. A separate 1,104-example diagnostic set (AX) is test-only, hidden-label, and scored by a three-class generalisation of Matthews correlation (R3) rather than folded into the nine-task average.\n",
   "splits": "9 tasks x (train, validation, test), MNLI's validation and test further split matched/mismatched; test labels hidden (returned as -1) for 8 of 9 tasks, public for MRPC; plus a 1,104-example test-only AX diagnostic set, scored separately",
   "url": "https://huggingface.co/datasets/nyu-mll/glue"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "lm_eval": "glue (tag grouping cola, mnli, mrpc, qnli, qqp, rte, sst, wnli as individually runnable tasks, each scored on the validation split; STS-B is not included in this tag)",
   "opencompass": "No single combined GLUE dataset was found; each task is configured separately, e.g. GLUE_CoLA, GLUE_MRPC, GLUE_QQP.",
   "other": "lm-evaluation-harness's own task README states that GLUE's official test-set labels are not publicly available, so it evaluates every subtask on the validation split instead of the hidden test split.\n"
  },
  "id": "glue",
  "last_updated": "",
  "leaderboard_url": "https://gluebenchmark.com/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "GLUE bundles nine separately scored English sentence- and sentence-pair-classification tasks behind one composite score, plus a hand-built diagnostic set reported on the side. Single-sentence tasks ask whether a sentence is grammatically acceptable (CoLA) or what sentiment it expresses (SST-2). Similarity and paraphrase tasks ask whether two sentences paraphrase each other (MRPC), how similar they are on a 1-5 scale (STS-B), or whether two Quora questions ask the same thing (QQP). Inference tasks recast entailment, reading comprehension and coreference as two- or three-way sentence-pair classification (MNLI, QNLI, RTE, WNLI). The nine tasks vary enormously in size by design, from a few hundred training pairs (WNLI) to hundreds of thousands (QQP, MNLI), so a model must generalise across data-scarce and data-rich tasks rather than simply having enough examples to fine-tune on any single one.\n",
  "metric": {
   "baseline_note": "No single random baseline applies across all nine tasks: they range from 0, the uninformed-guess value of the correlation-based metrics (CoLA, STS-B), to roughly 50-65% for the unbalanced binary classification tasks, so see each task's own page rather than the family average. The 87.1 human baseline is as reported in the SuperGLUE paper (Wang et al., 2019), which attributes it to Nangia and Bowman (2019); this page opened the SuperGLUE paper directly but did not independently open that underlying source.\n",
   "direction": "higher_is_better",
   "human_baseline": 87.1,
   "max_score": 100,
   "name": "GLUE Score: unweighted average of nine per-task scores (accuracy for five tasks, Matthews correlation for CoLA, mean of accuracy and F1 for MRPC and QQP, mean of Pearson and Spearman correlation for STS-B)",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "GLUE (General Language Understanding Evaluation benchmark)",
  "page_kind": "family",
  "paper": {
   "arxiv": "1804.07461",
   "title": "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding",
   "url": "https://arxiv.org/abs/1804.07461",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Alex Wang",
    "Amanpreet Singh",
    "Julian Michael",
    "Felix Hill",
    "Omer Levy",
    "Samuel R. Bowman"
   ],
   "org": "New York University",
   "url": "https://gluebenchmark.com/"
  },
  "released": "2018-04",
  "repo_url": "https://github.com/nyu-mll/GLUE-baselines",
  "saturation": {
   "as_of": "2019-07",
   "note": "The SuperGLUE paper records the GLUE state of the art as of early July 2019 at 88.4 (XLNet; Yang et al. 2019), 1.3 points above the 87.1 human-performance estimate it cites (Nangia and Bowman, 2019), and states the leading model exceeded that human estimate on four of the nine tasks outright. This page found no actively maintained current leaderboard reading beyond that point (see How to run it); today, frontier models are not commonly benchmarked on raw GLUE at all -- no model card in this repository reports a glue score, consistent with the field having moved to SuperGLUE and later suites.\n",
   "status": "saturated",
   "top_score": 88.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., arXiv:1804.07461)",
    "url": "https://arxiv.org/abs/1804.07461"
   },
   {
    "accessed": "2026-09-08",
    "title": "GLUE, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1804.07461"
   },
   {
    "accessed": "2026-09-08",
    "title": "nyu-mll/glue dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/nyu-mll/glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "nyu-mll/glue dataset API, Hugging Face",
    "url": "https://huggingface.co/api/datasets/nyu-mll/glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "nyu-mll/glue datasets-server info (split and config sizes)",
    "url": "https://datasets-server.huggingface.co/info?dataset=nyu-mll/glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "nyu-mll/GLUE-baselines GitHub repository",
    "url": "https://github.com/nyu-mll/GLUE-baselines"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (Wang et al., arXiv:1905.00537)",
    "url": "https://arxiv.org/abs/1905.00537"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1905.00537"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness glue tasks directory and README",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass GLUE_CoLA ppl config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/GLUE_CoLA/GLUE_CoLA_ppl_77d0df.py"
   }
  ],
  "status": "superseded",
  "subcategory": "multi-task English sentence and sentence-pair understanding suite (acceptability, sentiment, paraphrase, similarity, inference)",
  "summary": "A nine-task English sentence-understanding suite that defined pre-LLM benchmarking from 2018; models exceeded its human baseline within about 14 months, and its own successor SuperGLUE replaced it.",
  "tags": [
   "composite",
   "classification",
   "nlu",
   "superseded",
   "superglue",
   "historical"
  ],
  "task_format": "Nine separately trained-and-scored tasks, almost all single-sentence or sentence-pair classification; STS-B alone is a regression. Each task keeps its own metric (see How it is scored) and the nine scores are averaged, unweighted, into one GLUE Score out of 100. A separate, analysis-only diagnostic set (AX) is reported alongside the score but not folded into it.\n"
 }
}