SuperGLUE's word-sense task: decide whether a polysemous word has the same sense in two short sentences, scored by accuracy.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | binary word-sense disambiguation over sentence pairs |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 638 |
| Dataset licence | Hugging Face card licence is "other". SuperGLUE refers users to original dataset licences; a WiC-specific SPDX id was not stated on the SuperGLUE card or in the WiC paper HTML opened here. |
| Publisher | University of Cambridge / Tehran Institute for Advanced Studies and Cardiff University (original); SuperGLUE packaging at New York University |
WiC is word-sense disambiguation recast as a yes/no pair. The model sees two short English snippets and a target word that appears in both, then says whether that word is used with the same sense. Sentences come from WordNet, VerbNet and Wiktionary. SuperGLUE follows the original WiC paper and scores accuracy. The original authors stressed that most target words in the test split do not overlap the training vocabulary, so lexical memorisation is a weak strategy.
Binary classification over a sentence pair plus a marked word. lm-evaluation-harness uses a yes/no multiple-choice prompt. OpenCompass has a generation A/B config and several perplexity templates.
No model card in ModelSpec reports this benchmark yet.