An experimental HELM text-to-SQL scenario that asks a model to write a SQLite query, in English, over the classic 1999 Czech Bank relational dataset, graded only on whether the query runs.
unassessed
| Category | coding |
|---|---|
| Subcategory | text-to-SQL: generate a SQLite query from an English instruction against a fixed Czech banking relational schema |
| Page status | unknown |
| Metric | error_rate |
| Direction | lower_is_better |
| Unit | fraction |
| Dataset size | 102 |
| Publisher | Stanford CRFM (HELM), experimental scenario |
czech_bank_qa gives a model an English-language natural-language instruction plus a fixed eight-table SQLite schema (account, card, client, disp, district, loan, order, trans) drawn from the 1999 Czech Bank financial dataset, and asks it to output the single SQL query that would answer the instruction. The database itself holds Czech banking records, but the instructions, schema column names and expected output are all in English -- this is an English text-to-SQL task set against a Czech-sourced database, not a Czech-language benchmark.
Given the schema (embedded in the prompt) and an instruction, generate one SQL query with no training examples shown; HELM stops generation at a blank line and allows up to 512 tokens.
No model card in ModelSpec reports this benchmark yet.