{
 "body": "\n## What it measures\n\nDROP gives a model a short English passage -- commonly drawn from Wikipedia articles about NFL game\nrecaps, census data or history -- and a question that cannot be answered by finding one matching\nspan of text. A question typically requires resolving references to multiple positions in the\npassage and then performing a discrete operation over what is found there: adding or subtracting two\nnumbers, counting how many events satisfy a condition, sorting a set of values, or comparing two\ndates. Answers are a number, a date, or one or more short text spans copied from the passage; there\nare no multiple-choice options. It targets a specific gap the authors identified in earlier\nreading-comprehension datasets such as SQuAD: systems that had learned to locate a matching span\nwithout doing any arithmetic or comparison over what they found.\n\n## How it is scored\n\nPredictions are scored against a normalized reference answer with exact match (EM) and token-level\nF1, using number and date normalization so \"6\" and \"six\", or reordered multi-span answers, still\ncount as correct; the bipartite-matching approach for multi-span answers is adapted from SQuAD-style\nscoring. The original paper's own baselines set the scale: a semantic-parsing system reached 32.7%\nF1 and the authors' own NAQANet, combining a reading-comprehension model with simple numerical\nreasoning, reached 47.0% F1, against 96.0% F1 for expert human performance. lm-evaluation-harness\nand OpenCompass both report EM and F1; a fixed random-guess baseline is not meaningful for\nfree-response numeric, date and span answers.\n\n## Dataset and licence\n\nThe authors crowdsourced 96,567 question-answer pairs adversarially -- workers saw a passage and a\nbaseline model's live predictions, then wrote questions the model got wrong -- over roughly 7,000\nWikipedia passages, randomly partitioned by passage into training (80%), development (10%) and test\n(10%) so no passage's questions cross a split boundary. The public Hugging Face mirror (`ucinlp/drop`,\nCC BY-SA 4.0) carries only the training split (77,400 rows) and the development split, labelled\n`validation` (9,535 rows); the original test split is not in the public download, consistent with a\nleaderboard-graded evaluation whose test answers were never fully opened. In practice almost every\nharness, including lm-evaluation-harness, scores against this public validation split, so its answers\nare effectively public for anyone running the benchmark today. A second mirror lm-evaluation-harness\nitself uses, `EleutherAI/drop`, carries train (77,409) and validation (9,536) rows -- a small,\nunexplained difference from `ucinlp/drop`.\n\n## Who publishes it\n\nDROP was introduced by Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh and\nMatt Gardner, with affiliations spanning UC Irvine, the Allen Institute for AI, the University of\nWashington and the Hebrew University of Jerusalem, and published at NAACL 2019. AI2 hosts the dataset\nand reference evaluation code inside its `allennlp-reading-comprehension` repository and originally\nran a public leaderboard at leaderboard.allenai.org/drop; that URL did not respond during this\nresearch (2026-09), suggesting it may no longer be maintained. Today the benchmark is mainly kept\nusable through general-purpose harnesses -- lm-evaluation-harness, Inspect Evals and OpenCompass each\nship a `drop` task.\n\n## Lineage\n\nDROP has no formal predecessor or successor in this repository; it was built as a direct response to\nthe limits of span-extraction datasets like SQuAD, which the authors argued could be solved without\narithmetic or discrete reasoning. No dataset has explicitly superseded it, though its\nnumerical/discrete-reasoning framing anticipated a wave of later benchmarks that pair reading\ncomprehension with light computation.\n\n## Saturation and contamination\n\nDROP is not fully saturated, but the frontier moved well past its 2019 baselines. OpenAI's GPT-4\ntechnical report (2023-03) recorded 80.9 F1 3-shot -- ahead of GPT-3.5's 64.1 F1, but still short of\nQDGAT, a system trained specifically for DROP, at 88.4 F1, one of the few benchmarks in that report\nwhere a specialised model still beat GPT-4. That leaves real distance to the paper's 96.0%\nhuman-performance estimate, and no more recent frontier-model score was found during this research,\nso current standing is not established here. Contamination risk is medium: GPT-4's report separately\nestimated about 21% overlap between DROP and its pretraining data, but found GPT-4 scoring 82.5 F1 on\nthe non-overlapping subsample versus 80.9 F1 overall -- a higher score on the clean portion, evidence\nagainst contamination inflating that particular number. The public validation split's answers have\nbeen downloadable since 2019, so risk should be assumed higher for any model trained on more recent,\nless curated crawls.\n\n## How to run it\n\nlm-evaluation-harness's `drop` task (`generate_until`, scored on EM and F1 against the `EleutherAI/drop`\nvalidation split) implements the original AI2 evaluation logic from `allennlp-reading-comprehension`.\nInspect Evals and OpenCompass each ship their own `drop` task; Inspect Evals defaults to a 3-shot\nprompt. Because scoring depends on an answer-normalization step (numbers, articles, punctuation)\nrather than raw string match, reported EM and F1 can shift slightly between implementations that\nnormalize differently, and prompt format (zero-shot versus few-shot, whether the model is told where\nto stop generating) affects generation-based scoring more than it would a multiple-choice task.\n\n## Reading the numbers\n\nA high DROP score shows a model can combine simple arithmetic, counting or comparison with reading\ncomprehension, not just locate a matching sentence -- a meaningfully different skill from\nSQuAD-style extraction. Because the practically-used evaluation split has had public answers since\n2019, and frontier vendors rarely headline DROP scores today, treat any DROP number with the caution\ndue an older, unrefreshed benchmark: corroborate it against a newer numerical-reasoning benchmark\nbefore drawing conclusions about current capability. The gap AI2's own benchmark-specific baseline\nheld over general-purpose GPT-4 as late as 2023 is a reminder that a benchmark's ceiling and a\ngeneral model's score on it can diverge for a long time.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "drop",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "GPT-4's technical report separately estimates about 21% overlap between DROP and GPT-4's pretraining data, but reports GPT-4 scoring 82.5 F1 on the non-overlapping subsample versus 80.9 F1 overall -- a higher score on the clean portion, which argues against contamination inflating that particular result. The validation split's answers have been public since 2019 and are widely mirrored, so risk should be assumed higher for models trained on more recent, less curated web crawls without explicit decontamination.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 9535,
   "size_note": "96,567 question-answer pairs were collected in total (the paper's figure) over roughly 7,000 passages, randomly partitioned by passage into training (80%), development (10%) and test (10%) so no passage's questions cross a split. The public Hugging Face mirror (ucinlp/drop) carries only training (77,400 rows) and validation (9,535 rows, the number recorded here as `size` since it is the split almost every current harness actually scores against); the original held-out test split is not in the public download. A second mirror lm-evaluation-harness itself pulls from, EleutherAI/drop, carries train (77,409) and validation (9,536) -- a small, few-question difference from ucinlp/drop that neither dataset card explains.\n",
   "splits": "train (77,400), validation (9,535) on the public mirror; a held-out test split exists per the paper's 80/10/10 partition but is not in the public download",
   "url": "https://huggingface.co/datasets/ucinlp/drop"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "drop",
   "lm_eval": "drop",
   "opencompass": "drop",
   "other": ""
  },
  "id": "drop",
  "last_updated": "",
  "leaderboard_url": "https://leaderboard.allenai.org/drop/submissions/public",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "DROP gives a model a short English passage -- commonly a Wikipedia paragraph about an NFL game, a census report or a historical event -- and a question that cannot be answered by locating one matching span of text. Answering requires resolving references to multiple positions in the passage and then performing a discrete operation over what is found there: adding or subtracting numbers, counting how many events satisfy a condition, sorting values, or comparing dates. Answers are a number, a date, or one or more short text spans; there are no answer choices.\n",
  "metric": {
   "baseline_note": "The original paper reports 96.0% F1 for expert human performance and 32.7% F1 for the best 2019 system (a semantic-parsing baseline); the authors' own NAQANet model, combining reading comprehension with simple numerical reasoning, reached 47.0% F1. No meaningful random-guess baseline exists for free-response numeric, date and span answers.\n",
   "direction": "higher_is_better",
   "human_baseline": 96.0,
   "max_score": 100,
   "name": "F1 (exact match also reported)",
   "random_baseline": null,
   "unit": "% F1"
  },
  "name": "DROP",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1903.00161",
   "title": "DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs",
   "url": "https://arxiv.org/abs/1903.00161",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Dheeru Dua",
    "Yizhong Wang",
    "Pradeep Dasigi",
    "Gabriel Stanovsky",
    "Sameer Singh",
    "Matt Gardner"
   ],
   "org": "University of California, Irvine; Allen Institute for AI; University of Washington; Hebrew University of Jerusalem",
   "url": "https://allenai.org/data/drop"
  },
  "released": "2019-03",
  "repo_url": "https://github.com/allenai/allennlp-reading-comprehension",
  "saturation": {
   "as_of": "2023-03",
   "note": "OpenAI's GPT-4 technical report (2023-03) records 80.9 F1 3-shot on DROP, ahead of GPT-3.5's 64.1 but still behind QDGAT, a system trained specifically for DROP, at 88.4 F1 -- one of the few benchmarks in that report where a specialised model still beat GPT-4. That leaves real distance to the original paper's 96.0% human-performance estimate. No later frontier-model DROP score was found during this research, so current standing beyond 2023 is not established here.\n",
   "status": "watch",
   "top_score": 80.9
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (Dua et al., 2019)",
    "url": "https://arxiv.org/abs/1903.00161"
   },
   {
    "accessed": "2026-09-08",
    "title": "DROP paper, full text (ar5iv HTML)",
    "url": "https://ar5iv.labs.arxiv.org/html/1903.00161"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/allennlp-reading-comprehension GitHub repository (drop_eval.py reference scorer)",
    "url": "https://github.com/allenai/allennlp-reading-comprehension"
   },
   {
    "accessed": "2026-09-08",
    "title": "ucinlp/drop dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/ucinlp/drop"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/drop dataset (mirror used by lm-evaluation-harness)",
    "url": "https://huggingface.co/datasets/EleutherAI/drop"
   },
   {
    "accessed": "2026-09-08",
    "title": "DROP dataset homepage, Allen Institute for AI",
    "url": "https://allenai.org/data/drop"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness drop task",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/drop"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals drop task",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/drop"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass drop dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/drop"
   },
   {
    "accessed": "2026-09-08",
    "title": "GPT-4 Technical Report (OpenAI, 2023) -- DROP score and contamination estimate",
    "url": "https://arxiv.org/abs/2303.08774"
   }
  ],
  "status": "active",
  "subcategory": "reading comprehension with discrete reasoning",
  "summary": "An adversarially crowdsourced reading-comprehension test requiring numerical and discrete operations -- addition, counting, sorting -- over a paragraph, not just span lookup.",
  "tags": [
   "reading-comprehension",
   "numerical-reasoning",
   "discrete-reasoning",
   "free-response",
   "crowdsourced"
  ],
  "task_format": "Free-response reading comprehension: passage plus question in, a number, a date, or one or more text spans out; no answer choices."
 }
}