67 automatically generated 1,000-pair paradigms testing whether a model's probabilities favour the grammatical member of a minimal sentence pair -- linguistic knowledge, not task-solving ability.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | English grammatical acceptability, minimal-pair paradigms |
| Page status | active |
| Metric | pairwise accuracy (grammatical sentence assigned the higher probability) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 67000 |
| Dataset licence | CC BY 4.0 (per the Hugging Face nyu-mll/blimp dataset card) |
| Publisher | New York University |
BLiMP measures what a language model implicitly knows about English grammar, not whether it can perform a task. Each of its 67,000 items is a minimal pair: two sentences that differ by only one grammatical property -- for example subject-verb agreement, or whether a negative polarity item like "any" appears in a context that licenses it -- where one sentence is grammatical and the other is not. Rather than asking the model to answer a question or classify anything, BLiMP simply checks whether the model's own probability distribution assigns a higher likelihood to the grammatical sentence than to its ungrammatical, minimally different counterpart. This makes it a diagnostic of linguistic competence acquired implicitly during training, closer to a psycholinguistic acceptability-judgment experiment than to a benchmark of task-solving ability like question answering; a model can score well on BLiMP while being unable to explain, in words, why one sentence is grammatical and the other is not. The 67 paradigms group into 12 broader phenomena: anaphor agreement, argument structure, binding, control/raising, determiner-noun agreement, ellipsis, filler-gap dependencies, irregular forms, island effects, negative polarity item (NPI) licensing, quantifiers, and subject-verb agreement.
Forced-choice by probability comparison: for each of 67,000 minimal pairs, compare the log-probability the model assigns to a grammatical sentence against its minimally different ungrammatical counterpart; no explicit answer choice or generation is presented to the model.
No model card in ModelSpec reports this benchmark yet.