An adversarially crowdsourced reading-comprehension test requiring numerical and discrete operations -- addition, counting, sorting -- over a paragraph, not just span lookup.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | reading comprehension with discrete reasoning |
| Page status | active |
| Metric | F1 (exact match also reported) |
| Direction | higher_is_better |
| Unit | % F1 |
| Dataset size | 9535 |
| Dataset licence | CC BY-SA 4.0 |
| Publisher | University of California, Irvine; Allen Institute for AI; University of Washington; Hebrew University of Jerusalem |
DROP gives a model a short English passage -- commonly a Wikipedia paragraph about an NFL game, a census report or a historical event -- and a question that cannot be answered by locating one matching span of text. Answering requires resolving references to multiple positions in the passage and then performing a discrete operation over what is found there: adding or subtracting numbers, counting how many events satisfy a condition, sorting values, or comparing dates. Answers are a number, a date, or one or more short text spans; there are no answer choices.
Free-response reading comprehension: passage plus question in, a number, a date, or one or more text spans out; no answer choices.
No model card in ModelSpec reports this benchmark yet.