{
 "body": "\n## What it measures\n\nWiC asks whether a polysemous English word has the same sense in two short sentences. The model is given both sentences and the target word, and it answers yes or no. SuperGLUE describes this as word-sense disambiguation cast as binary classification. Sentences are drawn from WordNet, VerbNet and Wiktionary. The original paper (Pilehvar and Camacho-Collados, NAACL 2019) reports 5,428 / 638 / 1,400 train / dev / test instances, with no shared context sentences across splits and only 36% overlap of test target words with training.\n\n## How it is scored\n\nThe metric is accuracy. SuperGLUE Table 2 (test) gives most-frequent 50.0, BERT 69.5, and human 80.0. That human number is low relative to other SuperGLUE tasks: even annotators disagree on some sense pairs. Official SuperGLUE scoring uses hidden test labels. lm-evaluation-harness task `wic` scores the 638-row validation split with a yes/no multiple-choice prompt. OpenCompass `SuperGLUE_WiC` generation asks A/B (\"Are '{word}' in the above two sentenses the same?\") and scores accuracy after a first-capital postprocess; its perplexity configs compare \"same\" versus \"different\" continuations. Generation and perplexity numbers from that harness are not the same protocol.\n\n## Dataset and licence\n\nCounts used here are 5,428 / 638 / 1,400 from the WiC paper and from `aps/super_glue` config `wic`. SuperGLUE Table 1 rounds train to 6,000. The Hugging Face card licence is \"other\"; SuperGLUE does not publish a single SPDX id for WiC. Test labels are withheld for the official board. Harnesses opened for this page use the public validation split.\n\n## Who publishes it\n\nMohammad Taher Pilehvar and Jose Camacho-Collados introduced WiC (arXiv:1808.09121, 28 August 2018; NAACL 2019). SuperGLUE (Wang et al., May 2019, NeurIPS 2019) added it as one of eight tasks. The SuperGLUE site is super.gluebenchmark.com; it did not return scores as static HTML. The original GitHub README for WiC was not fetched (404 on the path tried).\n\n## Lineage\n\nWiC is not a GLUE task. SuperGLUE added it as a lexical-semantics check that GLUE lacked. This repository has no SuperGLUE family page. BoolQ (`boolq`) is a sibling SuperGLUE task with its own page. No WiC successor page is recorded here. The id is SuperGLUE's WiC, not an unrelated \"WIC\" acronym.\n\n## Saturation and contamination\n\nHuman accuracy on SuperGLUE WiC is 80.0, so a score in the high 70s is already near the published ceiling. BERT sat at 69.5 in the SuperGLUE paper. Later official tops were not readable. Combined with a public 638-row validation set from 2019, the task is treated as saturated. Contamination risk is high because the labelled validation items and the WordNet/Wiktionary sources have been easy to crawl for years.\n\n## How to run it\n\nlm-evaluation-harness: `wic` (tag `super-glue-lm-eval-v1`), dataset `aps/super_glue` config `wic`, validation accuracy. T5-prompt alias: `super_glue-wic-t5-prompt`. OpenCompass: directory `SuperGLUE_WiC`, abbreviation `WiC`, generation (`SuperGLUE_WiC_gen_d06864.py`) and several perplexity configs on `./data/SuperGLUE/WiC/val.jsonl`. Not confirmed in HELM, inspect_evals, or BIG-bench lists opened for this page. Name the prompt style (yes/no versus A/B versus ppl) before comparing two WiC accuracies.\n\n## Reading the numbers\n\nA high WiC accuracy means the model can often tell whether two short contexts share a sense of one English word. It does not measure multilingual WSD, token-level sense tagging, or definition generation. Because the published human number is 80, a model at 85 is past that estimate and may be fitting public items rather than solving a harder sense task. Use a newer lexical or multilingual sense benchmark if you need headroom among current models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "superglue_wic",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "WiC",
   "SuperGLUE_WiC",
   "wic"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "Train and validation labels have been public since the 2019 WiC and SuperGLUE releases. Source sentences are drawn from WordNet, VerbNet and Wiktionary, all widely copied into pretraining corpora. Harnesses reviewed here score the 638-row public validation split.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Hugging Face card licence is \"other\". SuperGLUE refers users to original dataset licences; a WiC-specific SPDX id was not stated on the SuperGLUE card or in the WiC paper HTML opened here.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 638,
   "size_note": "The original WiC paper Table 2 and Hugging Face `aps/super_glue` config `wic` agree on 5,428 train, 638 validation and 1,400 test instances. SuperGLUE Table 1 rounds train to 6,000 and keeps 638 / 1,400. This page uses the exact 5,428 / 638 / 1,400 counts. Official SuperGLUE scoring uses hidden test labels; harnesses score the 638-row validation split. The WiC paper states that only 36% of test target words overlap training, with no overlapping context sentences across splits.\n",
   "splits": "train 5,428 / validation 638 (public labels) / test 1,400 (official labels held out)",
   "url": "https://huggingface.co/datasets/aps/super_glue"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "wic (tag super-glue-lm-eval-v1; dataset aps/super_glue config wic; yes/no multiple choice on validation)",
   "opencompass": "SuperGLUE_WiC (abbr WiC; gen and several ppl configs on ./data/SuperGLUE/WiC/val.jsonl)",
   "other": "lm-evaluation-harness also ships super_glue-wic-t5-prompt under tag super-glue-t5-prompt."
  },
  "id": "superglue_wic",
  "last_updated": "",
  "leaderboard_url": "https://super.gluebenchmark.com/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "WiC is word-sense disambiguation recast as a yes/no pair. The model sees two short English snippets and a target word that appears in both, then says whether that word is used with the same sense. Sentences come from WordNet, VerbNet and Wiktionary. SuperGLUE follows the original WiC paper and scores accuracy. The original authors stressed that most target words in the test split do not overlap the training vocabulary, so lexical memorisation is a weak strategy.\n",
  "metric": {
   "baseline_note": "SuperGLUE Table 2 (test): most-frequent 50.0, BERT and BERT++ 69.5, human estimate 80.0. The original WiC paper (Pilehvar and Camacho-Collados, NAACL 2019) reports contextualised-embedding baselines in the high 50s to mid 60s on its own split, with BERT-large thresholding at 65.5, well below that SuperGLUE human figure.\n",
   "direction": "higher_is_better",
   "human_baseline": 80.0,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 50.0,
   "unit": "%"
  },
  "name": "SuperGLUE WiC (Word-in-Context)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1808.09121",
   "title": "WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations",
   "url": "https://arxiv.org/abs/1808.09121",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Mohammad Taher Pilehvar",
    "Jose Camacho-Collados"
   ],
   "org": "University of Cambridge / Tehran Institute for Advanced Studies and Cardiff University (original); SuperGLUE packaging at New York University",
   "url": "https://super.gluebenchmark.com/"
  },
  "released": "2019",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/super_glue/wic",
  "saturation": {
   "as_of": "",
   "note": "SuperGLUE's own human estimate is only 80.0 accuracy, with BERT already at 69.5 in 2019. The SuperGLUE leaderboard did not render as static HTML, so no later official top score is recorded. The ceiling is low and the validation items have been public since 2019, so the task is treated as saturated for frontier models.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "WiC: the Word-in-Context Dataset (Pilehvar and Camacho-Collados, arXiv:1808.09121)",
    "url": "https://arxiv.org/abs/1808.09121"
   },
   {
    "accessed": "2026-09-08",
    "title": "WiC full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1808.09121"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE (Wang et al., arXiv:1905.00537)",
    "url": "https://arxiv.org/abs/1905.00537"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperGLUE full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1905.00537"
   },
   {
    "accessed": "2026-09-08",
    "title": "aps/super_glue dataset card",
    "url": "https://huggingface.co/datasets/aps/super_glue"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness wic task config",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/super_glue/wic/default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass SuperGLUE_WiC generation config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/SuperGLUE_WiC/SuperGLUE_WiC_gen_d06864.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass SuperGLUE_WiC perplexity config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/SuperGLUE_WiC/SuperGLUE_WiC_ppl_312de9.py"
   }
  ],
  "status": "saturated",
  "subcategory": "binary word-sense disambiguation over sentence pairs",
  "summary": "SuperGLUE's word-sense task: decide whether a polysemous word has the same sense in two short sentences, scored by accuracy.",
  "tags": [
   "word-sense",
   "classification",
   "superglue",
   "saturated"
  ],
  "task_format": "Binary classification over a sentence pair plus a marked word. lm-evaluation-harness uses a yes/no multiple-choice prompt. OpenCompass has a generation A/B config and several perplexity templates.\n"
 }
}