{
 "body": "\n## What it measures\n\nLM-SynEval measures what a language model implicitly knows about specific points of English syntax,\nnot whether it can perform a task or explain a rule. Each item is a minimal pair of sentences\ndiffering in exactly one grammatical property -- for example \"The author laughs\" versus \"The author\nlaugh\" -- and the model is scored by whether it assigns a higher probability to the grammatical\nmember. The pairs cover three phenomena: subject-verb agreement (across a prepositional phrase, a\nsentential complement, relative clauses with and without an overt \"that\", and verb-phrase\ncoordination); reflexive anaphora (does a reflexive pronoun's number match its antecedent, within\nsimple sentences and across a relative clause); and negative polarity item licensing (does a word\nlike \"ever\" appear only where a context such as \"no\" licenses it). A model can score well on every\nconstruction here while being unable to state, in words, the rule it is implicitly satisfying -- this\nis a probe of linguistic competence, closer to a psycholinguistic acceptability-judgment experiment\nthan to a benchmark of task-solving ability such as question answering.\n\n## How it is scored\n\nThe model is scored correct on a pair if it assigns a higher probability to the grammatical sentence;\nlm-evaluation-harness implements this as a `multiple_choice` task with exactly two options per item,\nso random guessing scores 50%. Accuracy is reported per construction (72 total) and aggregates up\nthrough each phenomenon to an overall mean. The paper also recruited human participants online and\nreports \"a large gap remained\" between its best LSTM and human accuracy, though no single overall\nhuman-accuracy percentage was found in the source read for this page.\n\n## Dataset and licence\n\nThe Hugging Face mirror used by lm-evaluation-harness (`jmichaelov/lm_syneval`) totals 158,084\nsentence pairs across the 72 individual construction configs, confirmed via the Hugging Face\ndatasets-server. Per-construction size varies substantially, from 200 pairs for the smallest\nconstruction (long verb-phrase coordination) to over 2,800 for the largest (agreement across an\nobject relative clause). Sentences are automatically generated from linguist-built templates over a\nfixed vocabulary rather than drawn from a naturally occurring corpus. Both the original GitHub\nrepository and its Hugging Face mirror carry an MIT licence, and all pairs and their correct-answer\nlabels are public.\n\n## Who publishes it\n\nLM-SynEval was introduced by Rebecca Marvin and Tal Linzen in a paper presented at EMNLP 2018. The\noriginal reference implementation is hosted at github.com/BeckyMarvin/LM_syneval; no organisation\nruns an active public leaderboard for it, and lm-evaluation-harness maintains the version most\ncommonly used to evaluate current models.\n\n## Lineage\n\nLM-SynEval names no predecessor. This repository's `blimp` page (Warstadt et al., 2020, NYU) is the\nclearest downstream relative: BLiMP's own related-work table cites Marvin and Linzen 2018 by name for\nits coverage of subject-verb agreement, anaphor/binding and negative polarity items -- the same three\nphenomena this benchmark tests -- while BLiMP scales the same minimal-pair methodology to 67 paradigms\nand 12 phenomena with fully automatic generation. The two are separate benchmarks, not a family and a\nsubset of it, but they measure the same underlying kind of thing: probability-based grammaticality\njudgment on constructed minimal pairs.\n\n## Saturation and contamination\n\nThe original 2018 paper found an LSTM handled simple constructions well but performed poorly on\nharder ones, particularly agreement across an object relative clause and NPI licensing, and that\nmulti-task training with a CCG-supertagging objective improved but did not close the gap to human\naccuracy. No source opened for this page reported a current transformer-era or frontier-model score,\nand no model card in this repository cites it, so its standing against current models is not\nestablished here. Contamination risk is medium: the dataset has been public since 2018, but since\nevery pair is generated from a template over a shared vocabulary rather than hand-written, correctly\nfavouring grammatical sentences is plausibly learned from broad exposure to English generally, not\nfrom memorising these specific pairs.\n\n## How to run it\n\nlm-evaluation-harness implements the task as `lm_syneval`, made up of 72 construction tasks (for\nexample `lm_syneval__agreement__simple_agrmt__sing_MS_MV`) grouped under three sub-groups --\nagreement, reflexives, npi -- and an overall group, each using unweighted mean aggregation. The\nharness's own checklist notes this implementation has not been checked against the original authors'\nreference code, since that code targets RNN-family models rather than the prompted evaluation used\nfor current LLMs. No HELM, OpenCompass, inspect_evals or BIG-bench registration was confirmed.\n\n## Reading the numbers\n\nA high LM-SynEval score says a model's output probabilities are well calibrated to specific,\nnarrowly defined grammatical contrasts -- not that it can reason about, explain, or apply the\nunderlying rule in a downstream task, and not that it is broadly grammatical in open-ended\ngeneration. Because scores are reported per construction and phenomenon, an aggregate number can\nhide real unevenness: the original paper's own findings show some constructions (simple agreement)\nsolved far more easily than others (agreement across an object relative clause, NPI licensing), and\nthat pattern is exactly what the per-construction breakdown is for. Read this benchmark alongside\n`blimp` rather than in place of it: the two overlap in what they probe but differ in scale and\nconstruction set, so a strong score on one does not guarantee a strong score on the other.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lm_syneval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Targeted Syntactic Evaluation of Language Models",
   "Marvin and Linzen 2018"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "The dataset and its labels have been public since 2018, but every pair is generated automatically from linguist-built templates and a fixed vocabulary rather than hand-written or sourced from a single external corpus, so favouring grammatical sentences is a capability plausibly learned from broad exposure to grammatical English generally rather than from memorising these exact 158,084 pairs. This mirrors the reasoning this repository applies to the related `blimp` benchmark.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 158084,
   "size_note": "158,084 sentence pairs across 72 leaf task configs (confirmed via the Hugging Face datasets-server for the jmichaelov/lm_syneval mirror used by lm-evaluation-harness), grouped into 3 phenomena: agreement (the largest group, itself split across roughly 14 construction types such as simple agreement, agreement across a prepositional phrase, across a sentential complement, across subject/object relative clauses with and without \"that\", and verb-phrase coordination), reflexives (3 construction types) and negative polarity items (4 construction types, split by animacy and tense). Per-construction size varies widely, from 200 pairs (e.g. long verb-phrase coordination) to over 2,800 (e.g. agreement across an object relative clause).\n",
   "splits": "single 'test' split per construction; no train/validation split",
   "url": "https://huggingface.co/datasets/jmichaelov/lm_syneval"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 6, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "lm_syneval",
   "opencompass": "",
   "other": "The original authors' own repository (BeckyMarvin/LM_syneval) provides the reference generation and analysis code, built for evaluating RNN-family models; lm-evaluation-harness's own checklist for this task notes explicitly that it has not been checked against that reference implementation because the original pipeline targets a different (RNN) model architecture.\n"
  },
  "id": "lm_syneval",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "blimp"
   ],
   "variants": []
  },
  "measures": "LM-SynEval measures what a language model implicitly knows about specific points of English syntax, not whether it can perform a task or explain a rule. Each item is a minimal pair of sentences differing in one grammatical property -- for example \"The author laughs\" versus \"The author laugh\" -- and the model is scored by whether it assigns a higher probability to the grammatical member of the pair. The pairs are organised into three phenomena: subject-verb agreement (tested across many constructions -- across a prepositional phrase, a sentential complement, subject and object relative clauses with and without an overt \"that\", and verb-phrase coordination), reflexive anaphora (does the reflexive pronoun's number match its antecedent, again tested within simple sentences and across relative clauses), and negative polarity item licensing (does \"ever\" or a similar NPI appear only where a licensing context such as \"no\" makes it grammatical). A model can score well on every construction here while being unable to state, in words, the agreement or licensing rule it is implicitly satisfying -- this is a probe of linguistic competence, closer to a psycholinguistic acceptability experiment than to a benchmark of task-solving ability like question answering.\n",
  "metric": {
   "baseline_note": "Random guessing scores 50% on this binary forced choice (every lm-evaluation-harness item offers exactly the grammatical and ungrammatical sentence as its two choices). The original paper recruited human participants online and reports that \"a large gap remained\" between the best LSTM model's accuracy and human accuracy, but does not state one single overall human-accuracy percentage in the text read for this page, so no human_baseline figure is recorded here.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pairwise accuracy (grammatical sentence assigned the higher probability)",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "LM-SynEval (Targeted Syntactic Evaluation of Language Models)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1808.09031",
   "title": "Targeted Syntactic Evaluation of Language Models",
   "url": "https://arxiv.org/abs/1808.09031",
   "year": 2018
  },
  "publisher": {
   "authors": [
    "Rebecca Marvin",
    "Tal Linzen"
   ],
   "org": "",
   "url": "https://github.com/BeckyMarvin/LM_syneval"
  },
  "released": "2018-08",
  "repo_url": "https://github.com/BeckyMarvin/LM_syneval",
  "saturation": {
   "as_of": "",
   "note": "The original 2018 paper found that an LSTM language model handled simple constructions well but performed poorly on several harder ones (notably agreement across an object relative clause and NPI licensing), with multi-task training on CCG supertagging improving but not closing the gap to human accuracy. No source opened for this page reported a current transformer-era or frontier-LLM score on this specific benchmark, and no model card in this repository was found to cite it, so its standing against current models is not established here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Targeted Syntactic Evaluation of Language Models",
    "url": "https://arxiv.org/abs/1808.09031"
   },
   {
    "accessed": "2026-09-08",
    "title": "BeckyMarvin/LM_syneval repository (MIT licence)",
    "url": "https://github.com/BeckyMarvin/LM_syneval"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness lm_syneval README (phenomenon/construction breakdown with examples, citation, checklist)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lm_syneval/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness lm_syneval task template (multiple_choice format, doc_to_choice/doc_to_target)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lm_syneval/_template_yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "jmichaelov/lm_syneval dataset metadata (Hugging Face API) -- MIT licence",
    "url": "https://huggingface.co/api/datasets/jmichaelov/lm_syneval"
   },
   {
    "accessed": "2026-09-08",
    "title": "jmichaelov/lm_syneval datasets-server size endpoint (158,084 rows across 72 configs)",
    "url": "https://datasets-server.huggingface.co/size?dataset=jmichaelov/lm_syneval"
   },
   {
    "accessed": "2026-09-08",
    "title": "BLiMP (Warstadt et al. 2020, full text, ar5iv) -- related-work table citing Marvin and Linzen 2018 for the same three phenomena",
    "url": "https://ar5iv.labs.arxiv.org/html/1912.00582"
   }
  ],
  "status": "active",
  "subcategory": "targeted syntactic minimal pairs (agreement, reflexives, negative polarity items)",
  "summary": "72 auto-generated minimal-pair test sets probing whether a model's probabilities favour the grammatical sentence for subject-verb agreement, reflexive anaphora and negative polarity items.",
  "tags": [
   "linguistics",
   "grammar",
   "minimal-pairs",
   "agreement",
   "reflexives",
   "negative-polarity-items",
   "diagnostic",
   "probing"
  ],
  "task_format": "Forced-choice by probability comparison: for each minimal pair, compare the model's log-probability on the grammatical sentence against its ungrammatical, minimally different counterpart; the harness presents exactly two choices per item and no explicit answer or generation is requested.\n"
 }
}