{
 "body": "\n## What it measures\n\nBLiMP-NL asks whether a model\u2019s own probabilities prefer grammatical Dutch. Each item is a minimal pair: one well-formed sentence and one close ungrammatical match. The model is not prompted to classify or explain. Scoring only checks which sentence is more likely.\n\nThe 8,400 pairs (the large set) cover 84 paradigms grouped into 22 phenomena. Several of those phenomena are Dutch-specific, such as verb-second, R-word formation, and cross-serial verb clusters. This is not a translated copy of English [BLiMP](blimp.md).\n\n## How it is scored\n\nA pair is correct if the grammatical sentence has higher probability than the ungrammatical one. Random choice is 50%. The paper scores masked language models with SLOG, which adjusts for word frequency. EleutherAI lm-evaluation-harness cannot run masked models or SLOG, so it reports raw accuracy and byte-length-normalised accuracy. Those two numbers are not interchangeable with a published SLOG score.\n\nNative speakers judged every pair on a 7-point scale during self-paced reading. That study validates the contrasts. It is not a human run of the model likelihood task, so no human_baseline is recorded here.\n\n## Dataset and licence\n\nBLiMP-NL small has 10 hand-written pairs per paradigm (840 pairs). BLiMP-NL large expands each paradigm to 100 pairs. The extra 90 were generated with ChatGPT/GPT-3.5 Turbo, then checked by the authors. The Radboud deposit (DOI 10.34973/tj4p-y007) states CC BY-SA 4.0, created 2023-12-20 and published 2025-03-07. Hugging Face `jmichaelov/blimp_nl` restates CC-BY-SA-4.0 and ships one test TSV per paradigm in BLiMP column format (`sentence_good`, `sentence_bad`). There is no train split. Crossref lists the journal article itself as CC BY-NC-ND 4.0; that is not the dataset licence.\n\n## Who publishes it\n\nMichelle Suijkerbuijk and Stefan L. Frank (Radboud University, Centre for Language Studies) with Zo\u00eb Prins, Marianne de Heer Kloots and Willem Zuidema (University of Amsterdam, ILLC) released the corpus through Radboud\u2019s data repository. The article is in *Computational Linguistics* 51(4), 1267\u20131301 (DOI 10.1162/coli_a_00559). Crossref gives print and online as 2025-12-01; the lm-eval bibtex uses month 05. James Michaelov\u2019s Hugging Face mirror is the copy lm-eval loads. No maintained public leaderboard was found.\n\n## Lineage\n\nEnglish [BLiMP](blimp.md) is the design predecessor: templated grammatical minimal pairs scored by likelihood. BLiMP-NL is a new Dutch corpus, not a subset of that English file. The Radboud record also distinguishes the 840-pair small set from the 8,400-pair large set that lm-eval runs. No successor id exists in this repository.\n\n## Saturation and contamination\n\nNo current top-model number was confirmed here. The items have been public since 2025, so contamination is possible, especially for the generated majority of the large set. Favouring grammatical Dutch is also a skill a model can learn from ordinary text, which weakens the practical force of exact-pair leakage relative to unique exam items.\n\n## How to run it\n\nIn lm-evaluation-harness, `blimp_nl` runs all 84 tasks and macro-averages `acc` and `acc_norm`. Phenomenon groups such as `blimp_nl__wh_movement` average their paradigms the same way. Each YAML includes `_template_yaml`, uses dataset `jmichaelov/blimp_nl`, and compares the two sentences with an empty prompt. Do not treat an lm-eval `acc` figure as a reproduction of the paper\u2019s SLOG protocol.\n\n## Reading the numbers\n\nA high aggregate means the model ranks grammatical Dutch above matched ungrammatical variants, which is linguistic preference, not proof it can state a rule or translate. Read phenomenon scores; Dutch word order and filler-gap items can diverge from an English BLiMP profile. Compare only scores that used the same metric (`acc` vs `acc_norm` vs SLOG) and the same set (small 840 vs large 8,400).\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "blimp_nl",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "BLiMP-NL",
   "BLiMP-NL large"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "The large set and labels have been public since the 2025 Computational Linguistics paper and the Radboud deposit (DOI 10.34973/tj4p-y007, published 2025-03-07). Most pairs were model-generated then edited, so exact-string leakage is possible but the skill is also learnable from ordinary Dutch text. No memorisation study was opened here.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "nl"
   ],
   "license": "CC-BY-SA-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 8400,
   "size_note": "BLiMP-NL large: 84 paradigms \u00d7 100 pairs = 8,400. Ten pairs per paradigm were written by hand (840-pair small set); the other 90 were generated with ChatGPT/GPT-3.5 Turbo and then checked by the authors. lm-eval loads the large set from Hugging Face jmichaelov/blimp_nl (one TSV config per paradigm, test split only).\n",
   "splits": "test only; 84 paradigm configs, 100 pairs each",
   "url": "https://huggingface.co/datasets/jmichaelov/blimp_nl"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-029 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-029"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "blimp_nl",
   "opencompass": "",
   "other": "Group `blimp_nl` runs all 84 paradigm tasks and macro-averages acc and acc_norm (weight_by_size false). Phenomenon groups such as blimp_nl__verb_second sit under that group. Dataset path: jmichaelov/blimp_nl.\n"
  },
  "id": "blimp_nl",
  "last_updated": "2025-08",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "blimp",
   "successors": [],
   "variants": []
  },
  "measures": "BLiMP-NL tests whether a language model\u2019s probabilities favour grammatical Dutch over a minimally different ungrammatical sentence. Each item is a pair, not a question. The contrasts cover 22 syntactic phenomena that matter in Dutch, including verb-second order, R-words, crossing dependencies, and parasitic gaps, rather than a translation of English BLiMP. The model is not asked to label sentences or explain a rule. A high score means the distribution ranks the good sentence above the bad one.\n",
  "metric": {
   "baseline_note": "Chance is 50% on a binary pair. Native speakers rated every pair on a 7-point acceptability scale in a self-paced reading study; that is not the same as the model\u2019s probability comparison. No sourced figure for humans doing the likelihood task is recorded here. lm-eval reports un-normalised acc and byte-length-normalised acc_norm because it cannot run SLOG.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pairwise accuracy (grammatical sentence assigned the higher probability)",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "BLiMP-NL (Benchmark of Linguistic Minimal Pairs for Dutch)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "BLiMP-NL: A Corpus of Dutch Minimal Pairs and Acceptability Judgments for Language Model Evaluation",
   "url": "https://doi.org/10.1162/coli_a_00559",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Michelle Suijkerbuijk",
    "Zo\u00eb Prins",
    "Marianne de Heer Kloots",
    "Willem Zuidema",
    "Stefan L. Frank"
   ],
   "org": "Radboud University (Centre for Language Studies), with University of Amsterdam",
   "url": "https://data.ru.nl/collections/ru/cls/blimp-nl_dsc_550"
  },
  "released": "2025-03",
  "repo_url": "https://data.ru.nl/collections/ru/cls/blimp-nl_dsc_550",
  "saturation": {
   "as_of": "",
   "note": "No public leaderboard or current frontier-model score was confirmed from a source opened here. The paper appeared in 2025; saturation against today\u2019s models is not established.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BLiMP-NL paper (Computational Linguistics, DOI 10.1162/coli_a_00559)",
    "url": "https://doi.org/10.1162/coli_a_00559"
   },
   {
    "accessed": "2026-09-08",
    "title": "Radboud Data Repository: BLiMP-NL (dataset DOI 10.34973/tj4p-y007)",
    "url": "https://data.ru.nl/collections/ru/cls/blimp-nl_dsc_550"
   },
   {
    "accessed": "2026-09-08",
    "title": "jmichaelov/blimp_nl dataset card (CC-BY-SA-4.0, 8,400 pairs)",
    "url": "https://huggingface.co/datasets/jmichaelov/blimp_nl"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API metadata for jmichaelov/blimp_nl",
    "url": "https://huggingface.co/api/datasets/jmichaelov/blimp_nl"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness blimp_nl README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/blimp_nl/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval blimp_nl scoring template (acc and acc_norm)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/blimp_nl/_template_yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval blimp_nl group and phenomenon aggregates",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/blimp_nl/blimp_nl_group.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Crossref work 10.1162/coli_a_00559 (CL 51(4) 1267\u20131301; article CC BY-NC-ND 4.0)",
    "url": "https://api.crossref.org/works/10.1162/coli_a_00559"
   }
  ],
  "status": "active",
  "subcategory": "Dutch grammatical acceptability, minimal-pair paradigms",
  "summary": "8,400 Dutch minimal pairs across 84 paradigms and 22 phenomena, scored by whether a model prefers the grammatical sentence over a close ungrammatical match.",
  "tags": [
   "linguistics",
   "grammar",
   "minimal-pairs",
   "dutch",
   "diagnostic"
  ],
  "task_format": "Zero-shot forced choice by likelihood. lm-evaluation-harness leaves the prompt empty and compares log-probability of sentence_good against sentence_bad (doc_to_target 0). The paper\u2019s original scoring used masked models and syntactic log-odds ratios (SLOG); the harness does not implement SLOG.\n"
 }
}