HELM scenario that generates Dyck-n prefixes at run time and scores the unique closing-bracket suffix with exact match.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | generative Dyck-n closing-bracket completion |
| Page status | active |
| Metric | exact_match_indicator |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 500 |
| Dataset licence | Apache-2.0 |
| Publisher | Stanford Center for Research on Foundation Models (CRFM) |
HELM Dyck asks a model to finish a well-nested bracket prefix. The generator draws a Dyck-n string from a PCFG, then cuts it so the remaining suffix is only closing brackets. That suffix is unique for a given prefix. The task tests whether the model can track nested structure, not open-ended language. HELM's scenario can use one to four bracket pairs from (), [], {}, and <>. Completions are synthetic symbols, not natural English.
Completion. HELM prepends "Please complete the rest of the following Dyck sequences, making sure that the parentheses are closed properly." Inputs look like "( ( [". Outputs are space-separated closers. Default decoding: max_tokens 5, stop on newline, three in-context train items.
No model card in ModelSpec reports this benchmark yet.