{
 "body": "\n## What it measures\n\nSuperGLUE COPA asks a model which of two English alternatives is the more plausible cause or effect of a one-sentence premise. The question field is the word `cause` or `effect`. Items were written by hand for breadth and rater agreement, using photographic subject terms among other prompts, not harvested from tests. The original 2011 evaluation had 1,000 questions. SuperGLUE keeps that pool and reports it as one of eight suite tasks.\n\nThis is binary causal preference, not multi-hop explanation. A correct pick can come from shallow association as well as from a causal model.\n\n## How it is scored\n\nThe official SuperGLUE metric is accuracy. Random guessing is 50%. SuperGLUE Table 3 puts human accuracy at 100.0 on the hidden test set and BERT++ at 73.8. OpenCompass `SuperGLUE_COPA` generation configs use AccEvaluator and A/B postprocessing on `./data/SuperGLUE/COPA/val.jsonl` (100 labelled items). lm-evaluation-harness task `copa` uses the validation split of `aps/super_glue`/`copa`. Those harness scores are not the SuperGLUE test-server number. The original 2011 paper asked authors to publish both development and test accuracy; SuperGLUE hid the 500 test labels behind its server.\n\n## Dataset and licence\n\nOfficial SuperGLUE v2 files contain 400 train, 100 validation, and 500 test examples. Test jsonl omits `label`. Hugging Face `aps/super_glue` config `copa` matches those counts. Train labels in the zip are nearly balanced (195 vs 205) and mix cause and effect. The 2011 AAAI paper split the 1,000 questions equally into development and test (500 each) and asked authors to report both. SuperGLUE's 400/100 cut is that development half. The Hugging Face SuperGLUE card lists licence `other` and defers to original task licences. The 2011 AAAI paper carries a standard AAAI copyright line; no SPDX id was found on the SuperGLUE dump.\n\n## Who publishes it\n\nMelissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon introduced COPA at the 2011 AAAI Spring Symposium. SuperGLUE (Wang et al., arXiv:1905.00537, NeurIPS 2019) defined the 400/100/500 split and the leaderboard at super.gluebenchmark.com. OpenCompass's config directory `SuperGLUE_COPA` is the census spelling of that SuperGLUE task. The original ICT project page was not serving at the URL tried for this page; the AAAI PDF is the original-paper source used here.\n\n## Lineage\n\n`glue.md` is the predecessor suite. This repository already has `xcopa.md`, which translates and re-annotates English COPA validation and test items into 11 languages and is a different evaluation. Do not fold SuperGLUE COPA into XCOPA or the reverse. No standalone `copa.md` page exists here; this id is the SuperGLUE/OpenCompass spelling of English COPA. SuperGLUE's other tasks in this batch are `superglue_cb` and `superglue_multirc`; BoolQ is already `boolq.md`.\n\n## Saturation and contamination\n\nA 100% human ceiling on 500 hidden items, and a 100-item public validation file that harnesses score, leave little ranking room among current models. Contamination risk is high: the sentences have been public since 2011/2019 and are short enough to memorise. A 90% OpenCompass COPA number is not evidence of remaining causal-reasoning headroom.\n\n## How to run it\n\nOpenCompass: `opencompass/configs/datasets/SuperGLUE_COPA/` (generation and several PPL templates). Dataset abbr `COPA`; loader `COPADatasetV2`. lm-evaluation-harness: `--tasks copa` or `super_glue-copa-t5-prompt`. Official test accuracy needs the SuperGLUE server. inspect_evals and HELM had no COPA scenario in the sources opened here. Prompt templates differ (which may be the cause/effect vs T5 prefixes), so match configs before comparing.\n\n## Reading the numbers\n\nA strong SuperGLUE COPA score means the model usually picks the intended alternative on this small, old, binary set. It does not mean the model can explain the causal link, handle XCOPA's other languages, or beat chance on a hidden 500-item draw unless the reporter used the official test server. Prefer XCOPA or a newer commonsense suite when the claim is about causal reasoning rather than SuperGLUE completeness.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "superglue_copa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "COPA",
   "Choice of Plausible Alternatives",
   "SuperGLUE_COPA"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Train and validation labels have been public since SuperGLUE's 2019 release; the 2011 development questions are older still. Harnesses score the 100-row labelled validation split, not the hidden 500-row test file. Membership of these short, memorable sentences in pretraining data is plausible.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "other",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 1000,
   "size_note": "Official SuperGLUE v2 COPA.zip, counted from the jsonl files: 400 train (labels public), 100 validation (labels public), 500 test (labels omitted). Hugging Face `super_glue` config `copa` matches 400/100/500. The 2011 paper authored 1,000 questions and split them equally into development and test (500 each). SuperGLUE's 400/100 split of the original development portion is the usual train/val cut in modern harnesses.\n",
   "splits": "train 400 / validation 100 / test 500 (test labels withheld in the public files)",
   "url": "https://huggingface.co/datasets/aps/super_glue"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-002 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-002"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "copa",
   "opencompass": "SuperGLUE_COPA",
   "other": "lm-eval also ships super_glue-copa-t5-prompt; OpenCompass dataset abbr is COPA"
  },
  "id": "superglue_copa",
  "last_updated": "2025-05",
  "leaderboard_url": "https://super.gluebenchmark.com/",
  "lineage": {
   "family": "",
   "predecessor": "glue",
   "successors": [
    "xcopa"
   ],
   "variants": []
  },
  "measures": "SuperGLUE COPA is a two-choice causal commonsense task. The model reads one English premise sentence and a question that is either cause or effect, then picks which of two alternatives is more plausible. Items are hand-authored, not mined from exams. The original COPA paper (Roemmele, Bejan, and Gordon, 2011) wrote 1,000 such questions and split them 500/500 into development and test. SuperGLUE uses 400 train, 100 validation, and 500 test, matching the original test set and splitting the original development set. The language is English; the format is forced choice, not free-text explanation.\n",
  "metric": {
   "baseline_note": "Binary choice gives a 50% random-guess rate. SuperGLUE Table 3 estimates human accuracy at 100.0 on the hidden test set. BERT++ in that table scored 73.8. The original 2011 paper reported statistical NLP baselines well below that; those baselines are not HELM or OpenCompass numbers.\n",
   "direction": "higher_is_better",
   "human_baseline": 100,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "SuperGLUE COPA (Choice of Plausible Alternatives)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning",
   "url": "https://cdn.aaai.org/ocs/2418/2418-10878-1-PB.pdf",
   "year": 2011
  },
  "publisher": {
   "authors": [
    "Melissa Roemmele",
    "Cosmin Adrian Bejan",
    "Andrew S. Gordon",
    "Alex Wang",
    "Yada Pruksachatkun",
    "Nikita Nangia",
    "Amanpreet Singh",
    "Julian Michael",
    "Felix Hill",
    "Omer Levy",
    "Samuel R. Bowman"
   ],
   "org": "USC Institute for Creative Technologies and Indiana University (original COPA); SuperGLUE from New York University and collaborators",
   "url": "https://super.gluebenchmark.com/"
  },
  "released": "2019-05",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/SuperGLUE_COPA",
  "saturation": {
   "as_of": "",
   "note": "SuperGLUE's 2019 human estimate is 100% on the hidden test set, and the public validation split is 100 binary items. No current leaderboard table was recovered from the JavaScript SuperGLUE site, so no present-day machine top score is recorded. The combination of a perfect human ceiling, a tiny scored split, and 2011-era items is why COPA is treated as saturated inside SuperGLUE rather than as a ranking task.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems",
    "url": "https://arxiv.org/abs/1905.00537"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1905.00537"
   },
   {
    "accessed": "2026-09-08",
    "title": "Roemmele, Bejan, Gordon, AAAI Spring Symposium 2011 (COPA)",
    "url": "https://cdn.aaai.org/ocs/2418/2418-10878-1-PB.pdf"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official SuperGLUE v2 COPA.zip (400/100/500 jsonl)",
    "url": "https://dl.fbaipublicfiles.com/glue/superglue/data/v2/COPA.zip"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face super_glue dataset card (redirects to aps/super_glue)",
    "url": "https://huggingface.co/datasets/super_glue/resolve/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API: super_glue renamed to aps/super_glue; copa splits 400/100/500",
    "url": "https://huggingface.co/api/datasets/super_glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness copa task YAML",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/super_glue/copa/default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness SuperGLUE README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/super_glue/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass SuperGLUE_COPA generation config",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/SuperGLUE_COPA/SuperGLUE_COPA_gen_91ca53.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass COPADatasetV2 loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/copa.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE homepage (JavaScript app; scores not recovered as static text)",
    "url": "https://super.gluebenchmark.com/"
   }
  ],
  "status": "saturated",
  "subcategory": "binary causal commonsense: cause or effect of a premise (SuperGLUE)",
  "summary": "SuperGLUE packaging of COPA: pick the more plausible cause or effect of a one-sentence English premise from two alternatives.",
  "tags": [
   "causal-reasoning",
   "commonsense",
   "superglue",
   "classification",
   "saturated"
  ],
  "task_format": "Two-choice classification: premise plus cause/effect cue and two alternatives; English."
 }
}