An 8,000-item BIG-bench task built by inserting realistic speech disfluencies into SQuAD v2 questions, to test whether models stay robust when a question corrects or restarts itself.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | extractive reading comprehension under disfluent (self-corrected, restarted) questions |
| Page status | active |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 8000 |
| Dataset licence | CC BY 4.0 |
| Publisher | Google Research (submitted to the BIG-bench collaboration) |
disfl_qa asks a model to pick the correct answer span from a Wikipedia passage, given a question that has been rewritten to contain a natural contextual disfluency -- a correction or restart such as "What part [of] the composition of the Earth's biosphere is comprised of oxygen no sorry the Earth's atmosphere?" It measures robustness to a specific, understudied failure mode: whether shallow question-context matching breaks down once the question itself contains noise a fluent reader would filter out. Over 90% of DISFL-QA's disfluencies are corrections or restarts rather than simple filler, which the original paper's authors built specifically to be harder than earlier disfluency datasets that mostly repeated words.
Extractive question answering: given a disfluent question and a context passage, output the answer span (or "unknown" for unanswerable items); scored by exact_str_match against a single gold reference. A BIG-bench canary GUID is embedded in the task file.
No model card in ModelSpec reports this benchmark yet.