A programmatic BIG-bench task that matches masked SemCor contexts to WordNet definitions across 3,184 CoDA synset groups.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench programmatic alignment of masked SemCor contexts with WordNet definitions (3,184 groups) |
| Page status | unknown |
| Metric | alignment_score |
| Direction | higher_is_better |
| Dataset size | 3184 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
context_definition_alignment hides a noun or verb behind a nonce word (default bkatuhla) in a SemCor sentence and asks the model to pair that context with the matching WordNet 3.0 definition, one-to-one inside a synset group of size 5-10. Authors: Lütfi Kerem Şenel, Timo Schick, and Hinrich Schütze. English text. Programmatic task.py, not a JSON item list. Not ordinary word-sense disambiguation with the lemma given.
For each group the harness scores every context-definition pair by conditional log-probability of the definition after a pattern, then searches permutations for the best total. Preferred metric alignment_score (fraction of definitions matched). Four CoDA JSON files times nouns and verbs yield eight subtasks. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.