101-language minimal-pair benchmark of subject-verb agreement, scored by whether a model assigns higher probability to the grammatical sentence.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | massively multilingual subject-verb agreement minimal pairs |
| Page status | active |
| Metric | pairwise accuracy (grammatical sentence assigned the higher probability) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 121305 |
| Dataset licence | CC-BY-4.0 (Hugging Face card); Apache-2.0 (github.com/jumelet/multiblimp code) |
| Publisher | University of Groningen and Uppsala University |
MultiBLiMP 1.0 tests whether a language model's probabilities prefer a grammatical sentence over a minimally changed ungrammatical twin. Pairs are built automatically from Universal Dependencies treebanks and UniMorph inflections. The contrast is subject-verb agreement: two clause types (finite verb and participle) crossed with number, person and gender. That is six agreement conditions, which the v4 abstract also calls two types of subject-verb agreement. The model is not asked to label sentences. A high score means the distribution ranks the attested UD sentence above the inflected counterpart. Coverage is 101 languages, including several with only tens of pairs. It is not [blimp](blimp.md) (English, 67 hand-templated paradigms) and not [blimp_nl](blimp_nl.md) (Dutch).
Zero-shot forced choice by likelihood. lm-evaluation-harness leaves the prompt empty and compares log-probability of sen against wrong_sen (doc_to_target 0). The paper also reports a mean log- probability difference Δ. Hugging Face configs are one TSV per ISO 639-3 code; lm-eval sets test_split to train because those files have a single split.
No model card in ModelSpec reports this benchmark yet.