SuperGLUE's cloze reading-comprehension task: recover a masked entity in a CNN/Daily Mail query, scored by token-level F1 and exact match.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | cloze reading comprehension over news, with a masked named entity |
| Page status | saturated |
| Metric | max token-level F1 and exact match (EM), averaged for the SuperGLUE task score |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 10000 |
| Dataset licence | Hugging Face card licence is "other". The ReCoRD site says CNN/Daily Mail passages follow the Apache licence of the DeepMind rc-data dump, and Internet Archive crawls follow that archive's terms of use. |
| Publisher | New York University (SuperGLUE); original dataset from Johns Hopkins University and Microsoft |
ReCoRD asks a model to read a news passage and a cloze query in which one entity is replaced by a blank, then name the missing entity. Candidate answers are entities that already appear in the passage; several surface forms of the same entity all count as correct. The items come from CNN and Daily Mail articles. The original authors designed the queries so that many of them need commonsense about the article, not only span matching. SuperGLUE reports the task as English text-only reading comprehension and folds F1 and exact match into the suite average.
Cloze-style entity recovery given a passage, a query with a blank, and a list of in-passage entities. SuperGLUE and lm-evaluation-harness treat it as multiple-choice over those entities. OpenCompass instead generates an entity name and scores exact match against the gold aliases.
No model card in ModelSpec reports this benchmark yet.