{
 "body": "\n## What it measures\n\nSpider gives a model a natural-language question and the schema of a relational database -- tables,\ncolumns, types and foreign-key relationships -- and asks it to write the SQL query that answers the\nquestion. Its defining feature is that the databases in the train, development and test splits do\nnot overlap: a model cannot succeed by memorising a fixed database's query patterns and instead must\ngeneralise to schemas and domains it has never seen. The SQL queries themselves are complex, drawn\nfrom 138 different domains and spanning multiple tables, joins, nested subqueries, set operations\nand aggregation, not simple single-table lookups.\n\n## How it is scored\n\nTwo metrics are used, and they measure different things. Execution accuracy runs the model's\ngenerated query against the actual database and checks whether it returns the same result as the\ngold query, which tolerates structurally different but result-equivalent SQL. Exact set match\naccuracy instead decomposes the generated and gold queries into their SQL clauses (SELECT, WHERE,\nGROUP BY, and so on) and checks the sets match, ignoring literal values, which penalises structurally\ndifferent queries even if they would execute to the same result. The original 2018 paper's own best\nbaseline reached only 12.4% exact-match accuracy in the cross-domain (database-split) setting; no\nhuman baseline figure was found in the sources read for this page.\n\n## Dataset and licence\n\nSpider comprises 10,181 natural-language questions paired with 5,693 unique, human-written complex\nSQL queries across 200 databases spanning 138 domains, licensed CC BY-SA 4.0. Specific train,\ndevelopment and test split counts were not itemised in the sources read for this page, but the\nsplits are built so that no database appears in more than one split, the property that makes the\nbenchmark cross-domain. The original held-out test set was used for the official leaderboard, but\nSpider 1.0 stopped accepting new leaderboard submissions as of November 2024.\n\n## Who publishes it\n\nSpider comes from Tao Yu, Rui Zhang and ten co-authors at Yale University's LILY lab, led by\nDragomir Radev, published at EMNLP 2018. The Yale-LILY team maintains the official project page and\n(until its November 2024 closure) the public leaderboard.\n\n## Lineage\n\nSpider has no predecessor tracked in this repository; it explicitly built on and surpassed earlier,\nsingle-database text-to-SQL sets by introducing the cross-domain, multi-table design. Its authors\nhave since released Spider 2.0, a substantially harder successor aimed at real-world enterprise\ndatabase complexity, which does not yet have its own page in this repository.\n\n## Saturation and contamination\n\nThe official Spider 1.0 leaderboard closed to new submissions in November 2024. Its top execution-\naccuracy entry, \"MiniSeek\" (November 2023), reaches 91.2%; the system behind that entry is not\nidentified as GPT-4-based on the leaderboard. The best confirmed GPT-4-based entries -- DAIL-SQL\nwith GPT-4 and self-consistency, and DIN-SQL with GPT-4 -- reach 86.6% and 85.3% execution accuracy\nrespectively, still a large jump from the original 2018 baseline's 12.4% exact-match figure. That\ngap, together with the leaderboard's closure and the release of the harder Spider 2.0 successor,\npoints to the original benchmark being at or near effective ceiling for frontier-era systems.\nContamination risk is high: the train and development data, including their SQL queries, have been\nfreely downloadable since 2018 and are extremely widely used and cited, making them a plausible\ncomponent of many models' training data; the closure of the leaderboard also removes the practical\nprotection the held-out test set previously offered.\n\n## How to run it\n\nHELM implements it as the `spider` scenario, downloading the dataset, rendering a schema prompt from\nthe SQLite database files (sampling no rows by default) and scoring with an execution_accuracy\nmetric against the `test.json` split it uses. No lm-evaluation-harness, inspect_evals or OpenCompass\nimplementation was found for this page. Because execution accuracy depends on actually running\ngenerated SQL against the target database, and exact set match depends on how a harness normalises\nSQL clauses, scores from different harness implementations, or against execution accuracy versus\nexact match, are not directly comparable without checking which metric and database-access setup\nproduced them.\n\n## Reading the numbers\n\nA high Spider score indicates a model can translate natural-language questions into working SQL\nagainst database schemas it has not seen before, a meaningful measure of structured code generation\nand schema grounding rather than rote query memorisation, since the cross-domain split design was\nbuilt specifically to prevent that. Given the leaderboard's closure, its 91%+ top execution-accuracy\nentries, and the release of the harder Spider 2.0 successor, a near-ceiling score on original Spider\nsays less about a frontier model's current text-to-SQL ability than a low or mid-range score would;\ncontamination from Spider's widely reused public data is also a live concern for any model evaluated\non it today.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "spider",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Spider 1.0"
  ],
  "category": "coding",
  "contamination": {
   "note": "Spider's train and development splits, including their SQL queries and database contents, have been freely downloadable since 2018 and are among the most widely used and cited text-to-SQL resources in NLP research, making it likely that a large share of the public data is present in the training corpora of current large language models. The originally held-out test set was only accessible via leaderboard submission, but the leaderboard's closure as of November 2024 limits how much that held-out protection still matters for current evaluation.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": 10181,
   "size_note": "10,181 natural-language questions paired with 5,693 unique, human-written complex SQL queries, spanning 200 databases across 138 domains, confirmed from the original 2018 paper's own abstract and the official Yale-LILY project page. The databases and their schemas do not overlap between the train, development and (originally held-out) test splits, which is the benchmark's defining cross-domain design; specific train/dev/test counts were not itemised in the sources read for this page.\n",
   "splits": "Train, development and test; the original held-out test set was used for a public leaderboard, but as of November 2024 Spider 1.0 stopped accepting new leaderboard submissions",
   "url": "https://yale-lily.github.io/spider"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-004"
  },
  "harness": {
   "bigbench": "",
   "helm": "spider",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "spider",
  "last_updated": "2024-11",
  "leaderboard_url": "https://yale-lily.github.io/spider",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Spider gives a model a natural-language question and the schema of a relational database (table and column names, types and foreign keys) drawn from one of 138 domains, and asks it to produce the SQL query that answers the question. Unlike earlier text-to-SQL sets built on a single database, Spider's train/dev/test databases do not overlap, so a model must generalise to schemas and SQL structures it has never seen rather than memorise a fixed database's query patterns; the SQL queries themselves span multiple tables, joins, nesting, set operations and aggregation, not single-table lookups.\n",
  "metric": {
   "baseline_note": "The original 2018 paper's own best baseline model reached only 12.4% exact-match accuracy under the database-split (cross-domain) setting, illustrating how hard the cross-domain generalisation requirement was for contemporary systems; no explicit human baseline figure was found in the sources read for this page. Execution accuracy and exact set match can diverge because a query can produce the right result through a structurally different, non-gold SQL query, or the gold query itself can have several execution-equivalent phrasings.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "Execution accuracy (does the generated query, run against the database, return the same result as the gold query) and exact set match accuracy (does the generated query match the gold query's decomposed SQL clauses, ignoring literal values)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Spider",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1809.08887",
   "title": "Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task",
   "url": "https://arxiv.org/abs/1809.08887",
   "year": 2018
  },
  "publisher": {
   "authors": [
    "Tao Yu",
    "Rui Zhang",
    "Kai Yang",
    "Michihiro Yasunaga",
    "Dongxu Wang",
    "Zifan Li",
    "James Ma",
    "Irene Li",
    "Qingning Yao",
    "Shanelle Roman",
    "Zilin Zhang",
    "Dragomir Radev"
   ],
   "org": "Yale University (Yale-LILY lab)",
   "url": "https://yale-lily.github.io/spider"
  },
  "released": "2018-09",
  "repo_url": "https://github.com/taoyds/spider",
  "saturation": {
   "as_of": "2023-11",
   "note": "The official leaderboard (closed to new submissions since November 2024) lists \"MiniSeek\" as its top execution-accuracy entry at 91.2% (Nov 2023); the system behind that entry is not identified as GPT-4-based on the leaderboard itself. The best confirmed GPT-4-based entries -- DAIL-SQL with GPT-4 and self-consistency, and DIN-SQL with GPT-4 -- sit lower, at 86.6% and 85.3% execution accuracy respectively. All of these are well above the original 2018 baseline's 12.4% exact-match figure -- a large enough gap over five to six years to suggest the original benchmark is at or near effective ceiling for frontier-era systems. The project has since introduced Spider 2.0, a harder successor, consistent with the original benchmark being considered largely solved for top systems.\n",
   "status": "watch",
   "top_score": 91.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Yu et al., 'Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task' (EMNLP 2018 abstract: authors, dataset size, 12.4% baseline exact-match result)",
    "url": "https://arxiv.org/abs/1809.08887"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official Spider project page (Yale-LILY): dataset description, CC BY-SA 4.0 licence, November 2024 submission-closure notice, and leaderboard tables (top execution-accuracy entry MiniSeek 91.2% Nov 2023; top confirmed GPT-4-based entries DAIL-SQL and DIN-SQL at 86.6% and 85.3%)",
    "url": "https://yale-lily.github.io/spider"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM spider_scenario.py source (dataset download source, schema-prompt construction with num_rows=None, execution_accuracy metric)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/spider_scenario.py"
   }
  ],
  "status": "active",
  "subcategory": "cross-domain text-to-SQL semantic parsing: generate SQL from a natural-language question against an unseen database schema",
  "summary": "Spider is a large-scale, cross-domain text-to-SQL benchmark: a model must generate a correct SQL query for a natural-language question against one of 200 databases it has not seen during training.",
  "tags": [
   "text-to-sql",
   "code-generation",
   "semantic-parsing",
   "cross-domain"
  ],
  "task_format": "Given a database schema and a natural-language question, generate a syntactically and semantically correct SQL query. Reference harnesses such as HELM prompt the model with the schema (rendered from the SQLite database, table structure only, no example rows sampled by default) plus the question and, in HELM's implementation, a chain-of-thought instruction, expecting a single SQL query as output.\n"
 }
}