8,400 Dutch minimal pairs across 84 paradigms and 22 phenomena, scored by whether a model prefers the grammatical sentence over a close ungrammatical match.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | Dutch grammatical acceptability, minimal-pair paradigms |
| Page status | active |
| Metric | pairwise accuracy (grammatical sentence assigned the higher probability) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 8400 |
| Dataset licence | CC-BY-SA-4.0 |
| Publisher | Radboud University (Centre for Language Studies), with University of Amsterdam |
BLiMP-NL tests whether a language model’s probabilities favour grammatical Dutch over a minimally different ungrammatical sentence. Each item is a pair, not a question. The contrasts cover 22 syntactic phenomena that matter in Dutch, including verb-second order, R-words, crossing dependencies, and parasitic gaps, rather than a translation of English BLiMP. The model is not asked to label sentences or explain a rule. A high score means the distribution ranks the good sentence above the bad one.
Zero-shot forced choice by likelihood. lm-evaluation-harness leaves the prompt empty and compares log-probability of sentence_good against sentence_bad (doc_to_target 0). The paper’s original scoring used masked models and syntactic log-odds ratios (SLOG); the harness does not implement SLOG.
No model card in ModelSpec reports this benchmark yet.