Free-form question answering over a passage, where the questions form a real conversation and each answer needs the prior turns to be understood.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | conversational reading comprehension |
| Page status | saturated |
| Metric | F1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Mixed by source domain, as published on the CoQA website: CC BY-SA 4.0 (literature and Wikipedia passages), the MSR-LA licence (children's stories, from MCTest), RACE's own licence (middle/high school exam passages), and the Apache License (news passages, from the CNN/Daily Mail dataset). Two of the seven source domains (the paper's science and Reddit portions) are not released publicly at all. The Hugging Face dataset card tags the licence simply as "other." |
| Publisher | Stanford University |
CoQA tests whether a model can answer a series of interconnected questions about a passage, the way a person would in conversation, rather than one isolated question at a time. Passages are drawn from seven domains (children's stories, literature, middle and high school exams, news, Wikipedia, science and Reddit); each was given to two paired crowd-workers who chatted about it in questions and answers. Because later questions depend on earlier turns, CoQA requires coreference resolution (understanding what "it" or "she" refers to from prior turns) and pragmatic reasoning that single-turn reading-comprehension datasets do not exercise.
Free-form text answers to a sequence of conversational questions grounded in a single passage; each answer also comes with an evidence span highlighted in the passage. Systems are evaluated on an in-domain test set (domains seen in training) and an out-of-domain test set (domains held out entirely), scored against the official evaluation script.
No model card in ModelSpec reports this benchmark yet.