{
 "body": "\n## What it measures\n\nDeepSearchQA measures whether an LLM or web-connected agent can carry out a multi-step research task and\nreturn every correct piece of information, not just one plausible-sounding fact. Each of its 900 prompts\nspans one of 17 subject areas and requires a \"causal chain\" of lookups, where finding the answer to one\nstep depends on having correctly resolved the step before it. About two-thirds of the prompts expect a\nset of several correct answers rather than a single value.\n\nWhere earlier web-agent benchmarks mostly check whether an agent can find one correct fact, DeepSearchQA\nspecifically targets comprehensiveness: gathering scattered information across sources, de-duplicating\nand resolving entities correctly, and knowing when to stop searching rather than either quitting early or\npadding the answer with low-confidence guesses.\n\n## How it is scored\n\nGrading is outcome-based only: an agent's final answer set is compared against the gold answer by an\nautorater model, specified as gemini-2.5-flash running Google's own grading prompt from the official\nstarter notebook; the dataset card warns that a different judge model or prompt produces statistically\nsignificant deviation in results. Responses are bucketed as Fully Correct, Fully Incorrect, or Correct\nwith Excessive Answers (right answers padded with wrong ones), and F1 across those buckets is the metric\nthe official Kaggle leaderboard ranks by. Because grading is purely outcome-based, DeepSearchQA cannot\ndistinguish an agent that reasoned soundly from one that reached the right answer set through inefficient\nor lucky search.\n\n## Dataset and licence\n\nThe dataset holds 900 examples, each with a research prompt, a category label (one of 17 fields), a gold\nanswer, and an answer-type flag marking whether a single value or a set of values is expected; that flag\nis withheld from the model at inference time. It is released under Apache 2.0 by Google DeepMind, on\nHugging Face and Kaggle. Because every task is grounded in the live open web rather than a closed corpus,\nthe authors note that a task's ground truth can itself drift if a source page is edited or removed, which\nthey flag as a limitation requiring periodic manual review rather than a one-time release.\n\n## Who publishes it\n\nDeepSearchQA comes from a Google DeepMind team \u2014 Nikita Gupta, Riju Chatterjee, Lukas Haas, Connie Tao,\nAndrew Wang, Chang Liu, Hidekazu Oiwa, Elena Gribovskaya, Jan Ackermann, John Blitzer, Sasha Goldshtein\nand Dipanjan Das \u2014 posted to arXiv in January 2026 alongside a Google technical report, and announced on\nGoogle's developer blog together with the Gemini API's deep research agent tooling. DeepMind maintains\nthe official leaderboard and starter evaluation code on Kaggle.\n\n## Lineage\n\nDeepSearchQA does not name a direct predecessor in its own materials, but it sits in the same space as\nother agentic web-research benchmarks such as BrowseComp, which test different aspects of tool-using\nsearch agents; none of those has a page in this repository yet. No successor or subset benchmark has been\npublished under the DeepSearchQA name so far.\n\n## Saturation and contamination\n\nThe benchmark is new enough, and hard enough, that it has not saturated: the paper's own abstract reports\nthat even the most advanced agent architectures tested struggle to balance recall against precision,\nfailing in both directions, by stopping too early or by hedging with an overly wide net of low-confidence\nanswers. The official Kaggle leaderboard, last updated in December 2025, ranks Gemini Deep Research Agent\nfirst ahead of GPT-5 Pro, GPT-5.4, Gemini 3.1 Pro Preview, GPT-5, Claude Sonnet 4.6 and o3 Deep Research,\nthough this research could not confirm specific numeric scores from the page. Contamination risk is\ncurrently low: the benchmark was released in January 2026 and its answers are grounded in a live,\nchanging web rather than fixed text, which limits, though does not eliminate, straightforward\nmemorization from a static training snapshot.\n\n## How to run it\n\nThe dataset, technical report and starter grading code are distributed together: CSV data on Hugging Face\nand Kaggle, a technical report PDF hosted by Google, and a starter notebook on Kaggle implementing the\nrequired gemini-2.5-flash grading prompt. No lm-evaluation-harness, inspect_evals, HELM, OpenCompass or\nBIG-bench task was confirmed for it as of this research. Because the benchmark requires live web access\nand an LLM-based grader rather than exact-match scoring, reproducing a published number exactly depends\non matching the grading model and prompt, and on the target web pages not having changed since the\nreference run.\n\n## Reading the numbers\n\nA strong DeepSearchQA result shows an agent can plan a multi-step search, gather scattered facts from\nmultiple sources, and judge when its answer set is actually complete, a meaningfully different skill from\nsingle-hop question answering or fact lookup. It says nothing about the agent's efficiency, since grading\nis purely outcome-based, and nothing about the reasoning quality behind a lucky correct guess. Because\ngrading depends on a specific LLM judge and prompt, treat scores from different evaluation setups as not\ndirectly comparable, and prefer numbers taken from the official Kaggle leaderboard, which fixes the\ngrader.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "deepsearchqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Muse Spark",
   "model_id": "meta/muse-spark",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 74.8,
   "source": "meta-blog, officechai, artificial-analysis"
  }
 ],
 "page": {
  "aliases": [],
  "category": "agentic",
  "contamination": {
   "note": "Released January 2026, and every gold answer is grounded in the live open web rather than a fixed static corpus; the authors note ground truth can itself go stale if a source page is edited or removed, which limits straightforward memorization from a training snapshot but also complicates long-term reproducibility.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 900,
   "size_note": "900 prompts across 17 problem categories, each with a gold answer (single value or a set) and a category label.",
   "splits": "single eval split (900 rows)",
   "url": "https://huggingface.co/datasets/google/deepsearchqa"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Official grading requires the gemini-2.5-flash autorater with Google's own starter grading prompt (Kaggle starter notebook); the dataset card warns that a different judge model or prompt produces statistically significant deviation in results.\n"
  },
  "id": "deepsearchqa",
  "last_updated": "",
  "leaderboard_url": "https://www.kaggle.com/benchmarks/google/dsqa",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "DeepSearchQA measures whether an LLM or web-connected agent can carry out a multi-step research task and return every correct piece of information, not just one plausible-sounding fact. Each of its 900 prompts spans one of 17 subject areas and requires a \"causal chain\" of lookups, where finding the answer to one step depends on having correctly resolved the step before it. About two-thirds of the prompts expect a set of several correct answers rather than a single value, so the task specifically targets comprehensiveness and stopping judgment, not just single-hop retrieval.\n",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "F1 (over Fully Correct / Fully Incorrect / Correct with Excessive Answers)",
   "random_baseline": null,
   "unit": "score"
  },
  "name": "DeepSearchQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2601.20975",
   "title": "DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents",
   "url": "https://arxiv.org/abs/2601.20975",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Nikita Gupta",
    "Riju Chatterjee",
    "Lukas Haas",
    "Connie Tao",
    "Andrew Wang",
    "Chang Liu",
    "Hidekazu Oiwa",
    "Elena Gribovskaya",
    "Jan Ackermann",
    "John Blitzer",
    "Sasha Goldshtein",
    "Dipanjan Das"
   ],
   "org": "Google DeepMind",
   "url": "https://blog.google/technology/developers/deep-research-agent-gemini-api/"
  },
  "released": "2026-01",
  "repo_url": "",
  "saturation": {
   "as_of": "",
   "note": "The paper's own abstract reports that even the most advanced agent architectures tested struggle to balance recall against precision, failing by stopping too early or by hedging with an overly wide net of low-confidence answers. The official Kaggle leaderboard, snapshotted December 2025, ranks Gemini Deep Research Agent first ahead of GPT-5 Pro, GPT-5.4, Gemini 3.1 Pro Preview, GPT-5, Claude Sonnet 4.6 and o3 Deep Research, but this research did not confirm specific numeric scores from it.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents",
    "url": "https://arxiv.org/abs/2601.20975"
   },
   {
    "accessed": "2026-09-07",
    "title": "google/deepsearchqa dataset card",
    "url": "https://huggingface.co/datasets/google/deepsearchqa/blob/main/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "DeepSearchQA Leaderboard | Kaggle",
    "url": "https://www.kaggle.com/benchmarks/google/dsqa"
   }
  ],
  "status": "active",
  "subcategory": "deep research / web search agents",
  "summary": "900 multi-step web research tasks that grade an agent's full, deduplicated answer set rather than one fact.",
  "tags": [
   "agentic",
   "web-search",
   "deep-research",
   "long-horizon"
  ],
  "task_format": "Open-ended research prompt in; an LLM or agent with web access must return a complete, deduplicated set (or single value) of correct answers, graded by a separate autorater model against a gold answer.\n"
 }
}