{
 "body": "\n## What it measures\n\nTurBLiMP is the first Turkish benchmark of linguistic minimal pairs, modelled on BLiMP. Each item is a pair of Turkish sentences that differ by a single grammatical violation; a model that assigns higher probability to the grammatical sentence is credited as correct. The `turblimp_core` group covers 16 phenomena (agreement, argument structure, binding, ellipsis, island effects, nominalization, NPI licensing, passives, quantifiers, relative clauses, scrambling, and suspended affixation among them), with particular attention to Turkish's flexible word order and morphological subordination, which are understudied in existing minimal-pair benchmarks built for English.\n\n## How it is scored\n\nlm-evaluation-harness computes two variants: `acc`, based on the raw conditional log-probability of each sentence, and `acc_norm`, which normalizes by sentence length in bytes before comparing. Chance performance on the binary choice is 50%. The paper separately collected human acceptability judgments (30 native speakers rating 216 validation sentences on a 7-point Likert scale) to check that the intended grammatical sentence in each pair is in fact judged acceptable; this is a data-validation step, not a comparable accuracy baseline.\n\n## Dataset and licence\n\nThe `turblimp_core` group contains 16 phenomena with 1,000 minimal pairs each, for 16,000 pairs total, confirmed from the paper and the harness README. The source data repository (github.com/ezgibasar/TurBLiMP) is released under a CC BY 4.0 licence. The full TurBLiMP release also includes 20 additional experimental paradigms probing word order and subordination that sit outside the `turblimp_core` group and are not covered by this page.\n\n## Who publishes it\n\nTurBLiMP was introduced by Ezgi Ba\u015far, Francesca Padovani, Jaap Jumelet, and Arianna Bisazza, published at EMNLP 2025 (arXiv:2506.13487). EleutherAI's lm-evaluation-harness added the runnable task in pull request #3219, merged August 2025.\n\n## Lineage\n\nTurBLiMP follows the design of [BLiMP](blimp.md), the English benchmark of linguistic minimal pairs, extending the paradigm-based minimal-pair approach to Turkish's agglutinative morphology and flexible word order. No Turkish predecessor or successor benchmark was established.\n\n## Saturation and contamination\n\nThe paper reports that current large language models still struggle with several of the 16 phenomena despite these not being difficult for human speakers, so the benchmark still separates models; no current aggregate leaderboard was established. Contamination risk is assessed as medium: the benchmark is recent (June 2025) relative to the training cutoffs of many already-released models, but the sentence pairs are public and openly downloadable, so risk will rise for models trained after the release.\n\n## How to run it\n\nRun the lm-evaluation-harness group task `turblimp_core`, which aggregates the 16 phenomenon-specific tasks named `turblimp_<phenomenon>` (for example `turblimp_anaphor_agreement`, `turblimp_subject_agreement`). Report both `acc` and `acc_norm`, since normalization by length can change results for phenomena where the two sentences in a pair differ in length. Record the harness version, since the task was only added in August 2025.\n\n## Reading the numbers\n\nA strong score indicates a model reliably prefers the grammatical member of Turkish minimal pairs covering agreement, binding, word order, and subordination phenomena. It does not establish general Turkish fluency, production ability, or performance on the 20 additional experimental paradigms outside `turblimp_core`. Because scoring compares model log-probabilities rather than generated text, results depend on tokenization and are not directly comparable to classification-style Turkish benchmarks like TurkishMMLU. Compare `acc` against `acc_norm` and check the harness version used.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "turblimp_core",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "TurBLiMP"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "Released June 2025 (EMNLP 2025) and public on GitHub under CC BY 4.0; recent enough that inclusion in training data of already-released models is unlikely, but the sentence pairs are now openly downloadable.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "Turkish"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 16000,
   "size_note": "16 grammatical phenomena x 1,000 minimal pairs each in the turblimp_core group, per the paper and the harness README. The full TurBLiMP release adds 20 further experimental paradigms on word order and subordination that are outside this group.",
   "splits": "single evaluation set; no train/test split",
   "url": "https://github.com/ezgibasar/TurBLiMP"
  },
  "freshness": {
   "luna-new-002": null,
   "luna-new-002 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "turblimp_core",
   "opencompass": "",
   "other": ""
  },
  "id": "turblimp_core",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "blimp",
   "successors": [],
   "variants": []
  },
  "measures": "TurBLiMP tests whether a model's probabilities favour the grammatical member of a minimal pair of Turkish sentences, across 16 phenomena including subject and anaphor agreement, binding, island effects, scrambling, and suspended affixation, with particular attention to Turkish's flexible word order and morphological subordination.",
  "metric": {
   "baseline_note": "50% is chance for the binary sentence-pair choice. The paper separately collects human acceptability ratings (30 native speakers, 7-point Likert scale, 216 validation sentences) to validate item quality; that is not the same measure as accuracy and is not a directly comparable baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (acc) and length-normalized accuracy (acc_norm)",
   "random_baseline": 50,
   "unit": "percent"
  },
  "name": "TurBLiMP Core",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2506.13487",
   "title": "TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs",
   "url": "https://arxiv.org/abs/2506.13487",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Ezgi Ba\u015far",
    "Francesca Padovani",
    "Jaap Jumelet",
    "Arianna Bisazza"
   ],
   "org": "",
   "url": "https://github.com/ezgibasar/TurBLiMP"
  },
  "released": "2025",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/turblimp",
  "saturation": {
   "as_of": "",
   "note": "The paper reports that current LLMs still struggle with several of the 16 phenomena, which are not difficult for human speakers, so scores continue to separate models. No aggregate current leaderboard was established from the sources opened.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "TurBLiMP: A Turkish Benchmark of Linguistic Minimal Pairs (Ba\u015far, Padovani, Jumelet, Bisazza, EMNLP 2025)",
    "url": "https://arxiv.org/abs/2506.13487"
   },
   {
    "accessed": "2026-09-08",
    "title": "TurBLiMP data repository (license, dataset description)",
    "url": "https://github.com/ezgibasar/TurBLiMP"
   },
   {
    "accessed": "2026-09-08",
    "title": "Add TurBLiMP by jmichaelov, lm-evaluation-harness PR #3219 (merged)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/pull/3219"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness TurBLiMP task README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/turblimp/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness turblimp_core group config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/turblimp/turblimp_group.yaml"
   }
  ],
  "status": "active",
  "subcategory": "Turkish grammaticality, minimal-pair paradigms",
  "summary": "TurBLiMP's core group of 16 Turkish grammaticality phenomena, 1,000 minimal pairs each, run as a single lm-evaluation-harness task.",
  "tags": [
   "turkish",
   "linguistics",
   "classification",
   "minimal-pairs"
  ],
  "task_format": "Forced binary choice between two near-identical Turkish sentences that differ by one grammatical violation; the model is scored on which sentence it assigns higher log-probability to, not on a generated answer."
 }
}