SuperGLUE's recast Winograd Schema Challenge: decide whether a marked pronoun refers to a marked noun in one English sentence, scored by accuracy.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | binary pronoun coreference on Winograd schemas |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 104 |
| Dataset licence | Hugging Face card licence is "other". SuperGLUE refers users to original dataset licences; a WSC-specific SPDX id was not stated on the card opened here. |
| Publisher | New York University (SuperGLUE recast); original Winograd Schema Challenge from Levesque, Davis and Morgenstern |
SuperGLUE WSC is a binary coreference task, not the original fill-in-the-blank Winograd Schema Challenge. Each item is one English sentence with a marked pronoun and a marked noun. The model must say whether the pronoun refers to that noun. SuperGLUE built this recast from the original WSC items plus Commonsense Reasoning affiliated data, with a disjoint train/validation/test split so that GLUE WNLI's adversarial overlap cannot be memorised. The schemas are written to need everyday commonsense rather than syntax.
Binary yes/no classification on a sentence with two highlighted spans. lm-evaluation-harness loads Hugging Face config `wsc.fixed` and scores accuracy. OpenCompass has generation (A/B) and perplexity (Yes/No) configs on `val.jsonl`.
No model card in ModelSpec reports this benchmark yet.