A 250-trial BIG-bench task that asks which of four closing brackets finishes a Shuffle-4 prefix, testing counter-style counting rather than full Dyck hierarchy.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench Shuffle-4 last-closer multiple choice (250 trials) |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 250 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
dynamic_counting generates a Shuffle-n string (the shuffle of n Dyck-1 languages over distinct bracket pairs), drops the last symbol, and asks the model which closer belongs there. Default n is 4 over (), [], <>, {}. Authors Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov and Stuart M. Shieber argue that the last closer is the output of a real-time k-counter, not a stack that stores nested structure. The skill is counting opens and closes, not recognising a full Dyck language. It is not [dyck_languages](dyck_languages.md) (JSON Dyck-4 suffix) and not HELM [dyck_language](dyck_language.md).
Four-option multiple choice via cond_log_prob over the four closers. Preferred metric multiple_choice_grade. Zero-shot. Sequences drawn at evaluation time (seed 42, p=0.4, q=0.4, lengths 4-100, 250 unique trials). Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.