Text-to-SQL over 95 large databases; HELM's bird_sql run is the public 1,534-item development split scored by execution accuracy.
unassessed
| Category | coding |
|---|---|
| Subcategory | text-to-SQL on large, value-heavy databases |
| Page status | active |
| Metric | execution_accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1534 |
| Dataset licence | CC-BY-SA-4.0 |
| Publisher | Alibaba DAMO / HKU STAR Lab and collaborators (BIRD); Stanford CRFM (HELM scenario) |
BIRD-SQL (BIg Bench for Large-scale Database Grounded text-to-SQL) asks a model to write SQLite that answers an English question over a real-looking database, using both the schema and the stored values, plus a short evidence sentence of external knowledge. The full collection is 12,751 question–SQL pairs on 95 databases (33.4 GB) in 37 professional domains. HELM's scenario named bird_sql downloads the public development zip, builds a schema prompt plus the evidence comment, and asks for a chain-of-thought SQL answer in <sql> tags. It is not Spider and not the BIG-bench [semantic_parsing_spider](semantic_parsing_spider.md) wrap.
English question + schema dump + evidence to SQLite. HELM uses a tagged CoT prompt and execution-accuracy on the development databases. Official BIRD also reports Valid Efficiency Score (VES / R-VES) on the hidden test set, which HELM does not compute.
No model card in ModelSpec reports this benchmark yet.