Programmatic BIG-bench task that scores whether a model assigns higher probability to the grammatically correct verb form across syntactic constructions with long-distance or nested agreement.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | syntactic structure / linguistic probing |
| Page status | active |
| Metric | accuracy (mean condition score, 'full') |
| Direction | higher_is_better |
| Dataset size | 32116 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration), with data from four earlier academic papers |
This task probes whether a language model's internal representations capture the hierarchical (not just linear) structure of sentences, using subject-verb number agreement as the diagnostic. Given a sentence preamble such as 'The keys to the cabinet ...', the model must assign higher probability to the grammatically correct verb form (here, plural 'are') than to the form that would agree with a nearer but grammatically irrelevant noun ('cabinet', which would suggest singular 'is'). Constructions range from trivially adjacent subject-verb pairs to deeply nested relative clauses where multiple intervening nouns of conflicting grammatical number ('attractors') can mislead a model that relies on surface proximity rather than syntactic structure.
Two-way forced choice via log-probability comparison: for each item the model's conditional log-probability on the correct verb form is compared against the incorrect form, scored 1 if the correct form is more probable, 0 otherwise, averaged per condition and then across conditions/subtasks.
No model card in ModelSpec reports this benchmark yet.