{
 "body": "\n## What it measures\n\nLAMBADA gives a model a narrative passage -- several sentences of context followed by a final\nsentence with its last word removed -- and asks it to predict that missing word. The passages, drawn\nfrom a large collection of unpublished novels, were filtered by a two-stage human test: an annotator\nshown only the final sentence had to fail to guess the missing word, while an annotator shown the\nwhole passage had to succeed. That construction is deliberate: local, sentence-level statistics are\nnot enough, so a model has to track who is speaking, what was named earlier, and what situation the\nscene describes across the broader discourse to get the word right. It is English narrative text\nonly, evaluated as free next-word prediction rather than a multiple-choice task.\n\n## How it is scored\n\nModels are scored as next-token predictors: given the context up to the missing word, accuracy is\nthe fraction of passages where the model's highest-probability next token exactly matches the true\nword (perplexity over that same prediction is also commonly reported). There is no partial credit\nand no answer-choice list. The task has two harness variants that are not directly comparable:\n`lambada_standard` loads the original passages verbatim, while `lambada_openai` uses OpenAI's\nreformatted version of the same passages (introduced for the GPT-2 and GPT-3 papers, with light\ndetokenization differences); both keep the same 5,153-passage test count, but a model's accuracy can\ndiffer by a few points between them depending on tokenization sensitivity, so two reported LAMBADA\nnumbers are only comparable once you know which variant produced them.\n\n## Dataset and licence\n\nThe dataset consists of 10,022 narrative passages -- 4,869 for development and 5,153 for test --\ndrawn from 1,331 and 1,332 disjoint novels respectively, filtered from an initial 200,000 candidate\npassages judged by Mechanical Turk workers under the two-stage criterion above. A separate \"training\ndata\" release supplies the full text of 2,662 further novels (203 million words), disjoint from the\ndev/test novels, intended as background corpus for language models under test rather than additional\neval passages; the Hugging Face mirror's `train` split of 2,662 rows corresponds to these full\nnovels, not further cloze items. Hugging Face lists the dataset under a CC BY 4.0 licence; the\noriginal Zenodo archive (published 2016-08-07, coinciding with the paper's ACL 2016 presentation)\ndid not display a licence in the page content this research could read, so that reading is not\nindependently confirmed against the primary host.\n\n## Who publishes it\n\nLAMBADA was introduced by Denis Paperno, Germ\u00e1n Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham,\nRaffaella Bernardi, Sandro Pezzelle, Marco Baroni and Gemma Boleda at the University of Trento's\nCenter for Mind/Brain Sciences, with Raquel Fern\u00e1ndez at the University of Amsterdam, presented at\nACL 2016. The dataset is archived on Zenodo under the original authors' account rather than a\nmaintained leaderboard; no organisation currently runs a live LAMBADA leaderboard. It is kept usable\ntoday mainly through inclusion in general-purpose evaluation harnesses.\n\n## Lineage\n\nLAMBADA has no named predecessor or formal successor benchmark. It predates the modern\ninstruction-following and reasoning benchmark families by several years and belongs to an earlier\ngeneration of pure language-modelling evals, built to probe a specific linguistic phenomenon --\nlong-range discourse dependency -- rather than general task-solving ability. No variant or\ndescendant benchmark carrying the LAMBADA name has its own page in this repository.\n\n## Saturation and contamination\n\nLAMBADA saturated early relative to today's models. GPT-3's paper (Brown et al., May 2020) reported\n86.4% few-shot accuracy, an 8-point jump over the prior published state of the art (68.0%), with\nGPT-3's own zero-shot and one-shot settings at 76.2% and 72.5%. No later, larger-scale published\nnumber was found: the benchmark has largely fallen out of frontier system-card reporting since around\n2020, and no model card in this repository's own corpus currently carries a lambada score.\nContamination risk is high: the passages and answers have been public since 2016 and are widely\nmirrored, and GPT-3's own paper notes the dataset \"appears to be present\" in its training data, so\nany model trained on a broad web or books crawl since should be assumed to have seen these passages.\n\n## How to run it\n\nlm-evaluation-harness ships both variants under its `lambada` tag: `lambada_standard` (loading\n`cimec/lambada`) and `lambada_openai` (loading `EleutherAI/lambada_openai`, which also ships\ntranslated `de`/`es`/`fr`/`it` configs for a multilingual variant OpenAI produced). OpenCompass\ncarries its own `lambada` config. Both harness tasks score loglikelihood-based accuracy and\nperplexity rather than free generation, so results depend on exact tokenization and are sensitive to\nwhether a model's tokenizer splits the target word the same way the reference implementation\nexpects.\n\n## Reading the numbers\n\nA high LAMBADA score shows a model can maintain discourse-level context -- tracking a referent or\nscene across several sentences -- well enough to predict a final word that local statistics alone\nwould not give away; a model scoring near zero is failing at basic long-range coherence. Given the\ndataset's age, public availability since 2016, and near-universal presence in pretraining corpora, a\nstrong score today says very little about a model's current capability relative to its\ncontemporaries, and next to nothing about contamination-free understanding. Prefer a newer, actively\ncurated long-context or discourse-comprehension benchmark for any claim about a current model, and\ncheck whether a reported number used the `standard` or `openai` passage variant before comparing it\nto another.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lambada",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "LAMBADA dataset",
   "Word prediction requiring a broad discourse context"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Passages and their target words have been openly downloadable since 2016, and GPT-3's own paper explicitly states \"the LAMBADA dataset appears to be present in our training data,\" while arguing its own contamination analysis suggests limited score inflation from that overlap. Given the dataset's age and its ubiquity as a standard eval set mirrored across many corpora, assume any model trained on a broad web or books crawl after 2016 has likely seen these exact passages.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 5153,
   "size_note": "10,022 narrative passages total, split into 4,869 development and 5,153 test passages (the count recorded here), drawn from 1,331 and 1,332 disjoint novels respectively. Passages were filtered from an initial 200,000 candidates down to items every annotator answered correctly given the full passage and incorrectly given only the final sentence. A separate 2,662-novel, 203-million- word release provides full book text -- not further passages -- as background training corpus for language models under test, disjoint from the dev/test novels; the Hugging Face mirror's `train` split of 2,662 rows corresponds to these full novels. A reformatted `lambada_openai` variant (see How to run it) keeps the same 5,153-passage test count.\n",
   "splits": "train (2,662 rows, full novels used as background corpus, not eval passages), validation (4,869), test (5,153)",
   "url": "https://huggingface.co/datasets/cimec/lambada"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "lambada_standard, lambada_openai (grouped under a lambada tag)",
   "opencompass": "lambada",
   "other": ""
  },
  "id": "lambada",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "LAMBADA gives a model a narrative passage -- several sentences of context plus one final \"target\" sentence with its last word removed -- and asks it to predict that missing word. Passages were filtered so human readers shown only the final sentence could not guess the word, while readers shown the whole passage could. Succeeding therefore requires tracking information across the broader discourse -- who is speaking, what was named earlier, what the scene is -- rather than relying on local, sentence-level statistics, which is why the original paper frames it as testing \"genuine understanding of broad context.\"\n",
  "metric": {
   "baseline_note": "No fixed random baseline applies to open-vocabulary next-word prediction. The original 2016 paper reports that none of several contemporary language models exceeded 1% accuracy on the task at publication; no controlled human-accuracy figure is given beyond the two-stage filtering criterion used to build the dataset (full-passage guessable, final-sentence-only not guessable).\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (next-word exact match); perplexity also reported",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LAMBADA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1606.06031",
   "title": "The LAMBADA dataset: Word prediction requiring a broad discourse context",
   "url": "https://arxiv.org/abs/1606.06031",
   "year": 2016
  },
  "publisher": {
   "authors": [
    "Denis Paperno",
    "Germ\u00e1n Kruszewski",
    "Angeliki Lazaridou",
    "Quan Ngoc Pham",
    "Raffaella Bernardi",
    "Sandro Pezzelle",
    "Marco Baroni",
    "Gemma Boleda",
    "Raquel Fern\u00e1ndez"
   ],
   "org": "University of Trento (CIMeC); University of Amsterdam",
   "url": "https://zenodo.org/records/2630551"
  },
  "released": "2016-08",
  "repo_url": "https://zenodo.org/records/2630551",
  "saturation": {
   "as_of": "2020-05",
   "note": "The GPT-3 paper (Brown et al., 2020) reports 86.4% few-shot accuracy, calling it \"a gain of 8% over the previous state of the art\" (68.0%, zero-shot); GPT-3's own zero-shot and one-shot settings scored 76.2% and 72.5%. No model card in this repository's corpus carries a lambada score, and no more recent large-scale published number was found during this research, consistent with the benchmark having fallen out of frontier system-card reporting after around 2020 -- treat this as the most recent sourced reading rather than a current one.\n",
   "status": "saturated",
   "top_score": 86.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "The LAMBADA dataset: Word prediction requiring a broad discourse context (Paperno et al., 2016)",
    "url": "https://arxiv.org/abs/1606.06031"
   },
   {
    "accessed": "2026-09-08",
    "title": "LAMBADA paper, full text (ar5iv HTML)",
    "url": "https://ar5iv.labs.arxiv.org/html/1606.06031"
   },
   {
    "accessed": "2026-09-08",
    "title": "The LAMBADA dataset, Zenodo record (original release)",
    "url": "https://zenodo.org/records/2630551"
   },
   {
    "accessed": "2026-09-08",
    "title": "cimec/lambada dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/cimec/lambada"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/lambada_openai dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/EleutherAI/lambada_openai"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness lambada task (lambada_standard, lambada_openai)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/lambada"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass lambada dataset config",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/lambada"
   },
   {
    "accessed": "2026-09-08",
    "title": "Language Models are Few-Shot Learners (GPT-3 paper, Brown et al., 2020) -- LAMBADA results and contamination note",
    "url": "https://ar5iv.labs.arxiv.org/html/2005.14165"
   }
  ],
  "status": "active",
  "subcategory": "broad-context word prediction",
  "summary": "A last-word-prediction test built from narrative passages that humans can only complete correctly after reading the whole passage, not just the final sentence.",
  "tags": [
   "language-modelling",
   "cloze",
   "long-range-dependency",
   "zero-shot",
   "narrative"
  ],
  "task_format": "Language modelling / cloze: passage with its final word withheld in, a single predicted word out, scored as next-token prediction under a causal language model rather than multiple choice."
 }
}