HELM's eICU-only wrap of EHRSQL: write SQL for hospital questions, including unanswerable ones, and score execution accuracy.
unassessed
| Category | coding |
|---|---|
| Subcategory | clinical text-to-SQL on eICU with unanswerable questions |
| Page status | active |
| Metric | ehr_sql_execution_accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 12179 |
| Dataset licence | CC-BY-4.0 |
| Publisher | KAIST and collaborators (EHRSQL); Stanford CRFM (HELM scenario) |
This id is Stanford CRFM HELM's ehr_sql scenario, not the full two-database EHRSQL paper by itself. The model sees CREATE TABLE text from eICU plus an English clinical question and must emit SQL, or an empty string when the question cannot be answered from the schema. Questions come from a poll of 222 hospital staff. The original benchmark also covers MIMIC-III; HELM downloads only the eICU JSON and sqlite. It is not [BIRD-SQL](bird_sql.md).
Zero-shot SQL generation. HELM instructions require a query ending in semicolon, or an empty string for unanswerable items. max_tokens 1024, temperature 0, max_train_instances 0. An annotator executes predicted SQL on eicu.sqlite.
No model card in ModelSpec reports this benchmark yet.