{
 "body": "\nPart of the [GLUE](glue.md) family.\n\n## What it measures\n\nTwo English sentences, automatically pulled from online news sources, with a human annotation for whether\nthey are semantically equivalent -- a paraphrase judgement, not a similarity score like STS-B. The task is\nbinary sentence-pair classification: given both sentences, decide equivalent or not equivalent. The classes\nare imbalanced (68% positive, per the GLUE paper), which is why GLUE scores MRPC by the mean of accuracy\nand F1 rather than accuracy alone.\n\n## Reading the numbers\n\nA high MRPC score shows a model can recognise when two differently worded English news sentences describe\nthe same fact, a narrower skill than general similarity judgement and specific to short, formal news\nprose. Because the score averages accuracy and F1, a model can lift its number by improving either the\noverall hit rate or its balance between false positives and false negatives -- the two components are\nworth checking separately if a score looks unusual. MRPC is unusual among GLUE tasks in that its official\ntest-set labels are public, so, unlike CoLA or QQP, a \"test accuracy\" figure reported here can actually be\nverified against real labels rather than being a validation-split number in disguise. See the\n[GLUE](glue.md) family page for the benchmark's saturation history and contamination notes, which apply\nhere too.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "glue_mrpc",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "composite",
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Dataset card licence: other; see the glue family page for the composite-licence explanation.",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 408,
   "size_note": "408 validation rows (the split used for most published scoring), 3,668 train and 1,725 test rows. Unlike most other GLUE tasks, MRPC's Hugging Face test-split labels are real, publicly disclosed labels rather than -1 placeholders: the corpus already had a public test set (Dolan and Brockett, 2005) before GLUE adopted it, so GLUE did not need to hide it.\n",
   "splits": "train (3,668), validation (408), test (1,725, publicly labelled)",
   "url": "https://huggingface.co/datasets/nyu-mll/glue"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice B"
  },
  "harness": {
   "lm_eval": "mrpc (tag: glue; dataset nyu-mll/glue, config mrpc; scores acc and f1)",
   "opencompass": "GLUE_MRPC (evaluates both validation and test splits, unlike GLUE_QQP, since MRPC's test labels are genuinely public)"
  },
  "id": "glue_mrpc",
  "leaderboard_url": "https://gluebenchmark.com/leaderboard",
  "lineage": {
   "family": "glue"
  },
  "measures": "Two English sentences, automatically pulled from online news sources, with a human annotation for whether they are semantically equivalent -- a paraphrase judgement, not a similarity score. The task is binary sentence-pair classification: given both sentences, decide equivalent or not. The classes are imbalanced (68% positive, per the GLUE paper), which is why GLUE scores MRPC by the mean of accuracy and F1 rather than accuracy alone.\n",
  "metric": {
   "baseline_note": "No single random baseline is stated by the paper; with roughly 68% of examples labelled \"equivalent,\" always predicting the majority class scores well on accuracy alone, which is part of why F1 is averaged in alongside it. No task-specific human baseline is established here; see the glue family page.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "mean of accuracy and F1",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "GLUE: MRPC (Microsoft Research Paraphrase Corpus)",
  "page_kind": "subset",
  "paper": {
   "arxiv": "1804.07461",
   "title": "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding",
   "url": "https://arxiv.org/abs/1804.07461",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Alex Wang",
    "Amanpreet Singh",
    "Julian Michael",
    "Felix Hill",
    "Omer Levy",
    "Samuel R. Bowman"
   ],
   "org": "New York University",
   "url": "https://gluebenchmark.com/"
  },
  "released": "2018-04",
  "repo_url": "https://github.com/nyu-mll/GLUE-baselines",
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (Wang et al., arXiv:1804.07461)",
    "url": "https://arxiv.org/abs/1804.07461"
   },
   {
    "accessed": "2026-09-08",
    "title": "GLUE, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1804.07461"
   },
   {
    "accessed": "2026-09-08",
    "title": "nyu-mll/glue dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/nyu-mll/glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "nyu-mll/glue MRPC test-split rows (confirming public, non-dummy labels), Hugging Face datasets-server",
    "url": "https://datasets-server.huggingface.co/rows?dataset=nyu-mll/glue&config=mrpc&split=test&offset=0&length=20"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness glue/mrpc task config",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/glue/mrpc/default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass GLUE_MRPC ppl config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/GLUE_MRPC/GLUE_MRPC_ppl_96564c.py"
   }
  ],
  "status": "superseded",
  "subcategory": "paraphrase detection (sentence pair, English news)",
  "summary": "GLUE's paraphrase task: judge whether two English news sentences mean the same thing, scored by the mean of accuracy and F1 because the classes are unbalanced.",
  "tags": [
   "composite",
   "classification",
   "sentence-pair",
   "paraphrase",
   "glue-subset",
   "superseded"
  ],
  "task_format": "Binary sentence-pair classification (equivalent / not equivalent), commonly cast by harnesses as a two-way multiple-choice or yes/no generation task. Scored by the mean of accuracy and F1.\n"
 }
}