22 suites of 1,000 minimal pairs each that test whether models prefer grammatical Basque, Hindi, or Swahili continuations.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | targeted syntactic evaluation in Basque, Hindi, and Swahili |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 22000 |
| Publisher | Massachusetts Institute of Technology (authors); Hugging Face mirror by jmichaelov |
BHS is a targeted syntactic evaluation (TSE) for three lower-resource languages. Each item is a minimal pair: two sentences that differ by one word, only one of which is grammatical given the rest of the sentence. The model should assign higher probability to the grammatical member. Basque suites probe auxiliary agreement with subject, direct object and indirect object across several word orders. Hindi suites probe perfective versus non-perfective verb form with and without the ergative clitic ne, with optional possessors in between. Swahili suites probe noun-class agreement on verbs and adjectives across intervening material. Items are generated from vocabularies, not scraped corpora.
Zero-shot minimal-pair scoring. lm-eval implements each suite as a two-way multiple-choice task (ending_good vs ending_bad) with metrics acc and acc_norm. The authors' script compares length-normalised log-probability of the last word, which lm-eval cannot match exactly.
No model card in ModelSpec reports this benchmark yet.