{
 "body": "## What it measures\n\nWSC273 evaluates commonsense pronoun disambiguation using the first 273 sentence pairs of the Winograd Schema Challenge (WSC), proposed by Hector Levesque and expanded with Ernest Davis and Leora Morgenstern for KR-2012. Each schema is a pair of near-identical sentences that differ by one or two words, containing a pronoun whose correct referent flips between the two sentences depending on world knowledge, not grammar. The canonical example: \"The city councilmen refused the demonstrators a permit because they [feared/advocated] violence\" \u2014 \"they\" refers to the council if \"feared,\" to the demonstrators if \"advocated.\"\n\n## How it is scored\n\nThe lm-evaluation-harness `wsc273` task (`lighteval/winograd_wsc`, config `wsc273`) frames each item as a two-way multiple choice over which entity the pronoun refers to, scored as `acc` (accuracy) on the test split, with decontamination enabled using the full item text as the dedup query. Rather than asking the model to output a discrete choice, many implementations \u2014 following Trinh and Le's 2018 \"A Simple Method for Commonsense Reasoning\" \u2014 use partial evaluation: they substitute each candidate referent into the sentence and compare the language model's probability of the resulting continuation, picking the higher-probability substitution.\n\n## Dataset and licence\n\nThe dataset is the first 273 of a larger 285-sentence collection (WSC285) maintained on Ernest Davis's NYU page; researchers commonly report on the 273-item subset for consistency with earlier published results. The `lighteval/winograd_wsc` Hugging Face dataset, which the lm-evaluation-harness task loads, lists its licence as CC-BY-4.0 and contains 273 examples in its `wsc273` test split; there is no train or validation split.\n\n## Who publishes it\n\nThe Winograd Schema Challenge was proposed by Hector Levesque (Commonsense-2011, expanded for KR-2012) with Ernest Davis and Leora Morgenstern, and the schema collection is maintained on Davis's NYU page. EleutherAI's lm-evaluation-harness packages the 273-item subset as a runnable task; the dataset itself is also mirrored on Hugging Face by the `lighteval` project.\n\n## Lineage\n\nWSC273 is the widely-cited 273-item subset of the full WSC285 collection. [WinoWhy](winowhy.md), also in this repository, is built directly on the same 273 questions but asks for the justification behind each answer rather than the answer itself, and is recorded here as a successor task. [WinoGrande](winogrande.md) is a much larger, crowd-sourced benchmark in the same Winograd-schema style, but is a separate dataset rather than a direct successor to WSC273.\n\n## Saturation and contamination\n\nNo formal saturation study was found for current models, but WSC273 is widely regarded as easy for modern large language models compared to when it was introduced: Trinh and Le's 2018 language-model scoring method already reached well above the human-competitive threshold the original challenge targeted. Because the 273 sentences and their correct answers have been publicly available since 2011-2012 and are reproduced across many papers, blog posts and other evaluation repositories, contamination risk is high for models trained on large web crawls.\n\n## How to run it\n\nRun the `wsc273` task in lm-evaluation-harness, which uses the `lighteval/winograd_wsc` dataset (config `wsc273`) and reports mean accuracy. Confirm whether a given implementation scores via explicit multiple-choice selection or via language-model partial evaluation (the Trinh and Le method), since the two protocols can produce different scores for the same model.\n\n## Reading the numbers\n\nA high WSC273 score shows a model resolves classic pronoun-ambiguity puzzles that require basic world knowledge, but because the item set is small, old, and highly reproduced online, near-ceiling scores from recent large models are hard to distinguish from memorisation. Compare a WSC273 score alongside [WinoWhy](winowhy.md), which tests whether the model can justify the answer rather than just produce it, and alongside a larger, less-contaminated set such as [WinoGrande](winogrande.md) before treating a high score as evidence of general commonsense reasoning.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "wsc273",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "The 273 sentences and their answers have been public since 2011-2012 and are widely reproduced in papers, blog posts and other benchmark repositories, making verbatim memorisation plausible for models trained on large web crawls.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "cc-by-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 273,
   "size_note": "273 Winograd schema sentence pairs, the first 273 items of the larger WSC285 (285-sentence) release, used for consistency with earlier reported results.",
   "splits": "test only (273 examples)",
   "url": "https://huggingface.co/datasets/lighteval/winograd_wsc"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "wsc273",
   "opencompass": "",
   "other": ""
  },
  "id": "wsc273",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "winowhy"
   ],
   "variants": []
  },
  "measures": "WSC273 evaluates commonsense pronoun disambiguation on the first 273 items of the Winograd Schema Challenge, a set of near-identical sentence pairs whose correct pronoun referent flips on world knowledge alone.",
  "metric": {
   "baseline_note": "Two-way choice gives a 50% random baseline; not restated as a fixed number in the harness config itself.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 50,
   "unit": "percent"
  },
  "name": "WSC273",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "The Winograd Schema Challenge",
   "url": "https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html",
   "year": 2012
  },
  "publisher": {
   "authors": [
    "Hector Levesque",
    "Ernest Davis",
    "Leora Morgenstern"
   ],
   "org": "",
   "url": "https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html"
  },
  "released": "2011",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/wsc273",
  "saturation": {
   "as_of": "",
   "note": "",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "wsc273 default.yaml task config (lm-evaluation-harness)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/wsc273/default.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "wsc273 task README (lm-evaluation-harness)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/wsc273/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lighteval/winograd_wsc dataset card metadata",
    "url": "https://huggingface.co/api/datasets/lighteval/winograd_wsc"
   },
   {
    "accessed": "2026-09-08",
    "title": "The Winograd Schema Challenge (official collection page)",
    "url": "https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "A Simple Method for Commonsense Reasoning (Trinh & Le, 2018)",
    "url": "https://arxiv.org/abs/1806.02847"
   }
  ],
  "status": "active",
  "subcategory": "coreference resolution",
  "summary": "WSC273 scores pronoun-resolution accuracy on the first 273 items of the Winograd Schema Challenge, using language-model probability rather than fine-tuning.",
  "tags": [
   "benchmark",
   "commonsense",
   "coreference"
  ],
  "task_format": "Multiple-choice: given a sentence with an ambiguous pronoun and two candidate referents that differ by one or two words from a paired sentence, the model must pick the correct referent, scored via language-model probability of the completion (partial evaluation) rather than explicit answer selection."
 }
}