Spider is a large-scale, cross-domain text-to-SQL benchmark: a model must generate a correct SQL query for a natural-language question against one of 200 databases it has not seen during training.
unassessed
| Category | coding |
|---|---|
| Subcategory | cross-domain text-to-SQL semantic parsing: generate SQL from a natural-language question against an unseen database schema |
| Page status | active |
| Metric | Execution accuracy (does the generated query, run against the database, return the same result as the gold query) and exact set match accuracy (does the generated query match the gold query's decomposed SQL clauses, ignoring literal values) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 10181 |
| Dataset licence | CC BY-SA 4.0 |
| Publisher | Yale University (Yale-LILY lab) |
Spider gives a model a natural-language question and the schema of a relational database (table and column names, types and foreign keys) drawn from one of 138 domains, and asks it to produce the SQL query that answers the question. Unlike earlier text-to-SQL sets built on a single database, Spider's train/dev/test databases do not overlap, so a model must generalise to schemas and SQL structures it has never seen rather than memorise a fixed database's query patterns; the SQL queries themselves span multiple tables, joins, nesting, set operations and aggregation, not single-table lookups.
Given a database schema and a natural-language question, generate a syntactically and semantically correct SQL query. Reference harnesses such as HELM prompt the model with the schema (rendered from the SQLite database, table structure only, no example rows sampled by default) plus the question and, in HELM's implementation, a chain-of-thought instruction, expecting a single SQL query as output.
No model card in ModelSpec reports this benchmark yet.