{
 "body": "\n## What it measures\n\nThis group asks a language model to predict the last word of a short narrative\npassage after seeing the rest of the text. The construction is the OpenAI\nreformat of English [LAMBADA](lambada.md): humans could guess the word from the\nfull passage, not from the last sentence alone. Four of the five configs are\nmachine translations of that English test split into German, Spanish, French and\nItalian. The fifth is English under Hugging Face config `en`. The intended skill\nis still discourse tracking, but a wrong translation of the target word can fail\nthe item even when the model tracked the story.\n\n## How it is scored\n\nEach YAML task uses loglikelihood scoring. Accuracy is the fraction of passages\nwhere the model's next-token ranking matches the gold last word; perplexity is\nalso reported and is lower-is-better. There is no answer list and no partial\ncredit. Tokenization of the gold word matters, as on English LAMBADA. The\nrunnable names are `lambada_openai_mt_en` and the four `lambada_openai_mt_{de,es,fr,it}`\ntasks; the group name `lambada_multilingual` runs all five. Do not mix these\nnumbers with `lambada_openai` or `lambada_standard`.\n\n## Dataset and licence\n\nEleutherAI/lambada_openai publishes five language configs plus `default`, each\nwith 5,153 test rows on the Hugging Face card and API (created 2022-12-16). The\nnon-English files were written with googletrans from `lambada_test_en.jsonl` by\nSid Black; the script in the dataset repo names Google Translate front-ends.\nThe card's YAML `license` field is MIT. The same README's Licensing section\ninstead says \"Modified MIT\" and links OpenAI's GPT-2 licence. That is a\ndisagreement on one card, not a second dataset. Original `cimec/lambada` English\nis documented as CC BY 4.0 on [lambada](lambada.md). This page records the\nEleutherAI mirror the harness loads, with both MIT and Modified MIT readings.\n`default` and `en` report the same 5,153 rows and the same byte size on the\nHugging Face API; they were not hashed against each other here.\n\n## Who publishes it\n\nThe passages come from Paperno and colleagues' 2016 LAMBADA set. OpenAI's GPT-2\nera reformat is the English source for this mirror. EleutherAI hosts the dataset\nand the harness group. The mt YAML files were added to lm-evaluation-harness on\n2023-07-05. No organisation runs a dedicated multilingual LAMBADA leaderboard.\n\n## Lineage\n\nPredecessor is English [LAMBADA](lambada.md). This page is the harness group,\nnot a spelling of `lambada_openai`. A separate directory\n`lambada_multilingual_stablelm` exists in the harness and has no page here.\nMachine translation is a change of language, not a new discourse filter, so a\nstrong German score is not evidence that the LAMBADA human filter still holds.\n\n## Saturation and contamination\n\nNo current top score for the five mt tasks was read from a leaderboard. English\nLAMBADA was already high by 2020. Contamination risk is high: the English items\nare old and public, and the translations have been downloadable since December\n2022. The YAML enables decontamination queries over the full passage; that only\nhelps pipelines that actually filter.\n\n## How to run it\n\nIn lm-evaluation-harness, run the group `lambada_multilingual` or a single\n`lambada_openai_mt_*` task. Each loads `EleutherAI/lambada_openai` with\n`dataset_name` set to `en`, `de`, `es`, `fr` or `it`, `test_split: test`.\nScoring is loglikelihood, not generate-until. OpenCompass and HELM names for\nthis group were not confirmed. Compare two numbers only when they share a\nlanguage config and the OpenAI-format detokenization.\n\n## Reading the numbers\n\nA high score on one language means the model assigned the gold last word the\nbest likelihood on that 5,153-row split. On de/es/fr/it it can also mean the\ntranslation left a locally predictable last word, or that the model memorised\nthe English story and the translated target. The English mt task may overlap\n`lambada_openai` on [lambada](lambada.md). Prefer a native, human-filtered\ncloze if the claim is about discourse in that language, and say which of the\nfive tasks produced the number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lambada_multilingual",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "lambada_openai_mt",
   "lambada_mt"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The OpenAI-format English split has been public for years and is widely mirrored. The four translations have been on Hugging Face since the dataset was created on 2022-12-16 (card lastModified 2025-07-10). YAML sets should_decontaminate: true with the full passage as the query. Machine translation does not make the items private.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en",
    "de",
    "es",
    "fr",
    "it"
   ],
   "license": "MIT (Hugging Face cardData); README Licensing section cites Modified MIT (OpenAI GPT-2)",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 25765,
   "size_note": "Five Hugging Face configs on EleutherAI/lambada_openai, each with a test split of 5,153 examples (en, de, es, fr, it), totalling 25,765 scored passages if the group is run in full. The card's `default` config is also 5,153 rows and is what lambada_openai uses; this page did not byte-compare `default` with `en`. Translations were produced with googletrans (Google Translate) by Sid Black, from lambada_test_en.jsonl.\n",
   "splits": "test only, 5,153 rows per language config",
   "url": "https://huggingface.co/datasets/EleutherAI/lambada_openai"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-011 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-011"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "lambada_multilingual",
   "opencompass": "",
   "other": "Runnable tasks lambada_openai_mt_en, lambada_openai_mt_de, lambada_openai_mt_es, lambada_openai_mt_fr, lambada_openai_mt_it. Group tag lambada_multilingual. Distinct from lambada_openai / lambada_standard under the lambada group, and from lambada_multilingual_stablelm."
  },
  "id": "lambada_multilingual",
  "last_updated": "2025-07",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "lambada",
   "successors": [],
   "variants": []
  },
  "measures": "lambada_multilingual is EleutherAI's lm-evaluation-harness group over five OpenAI-format LAMBADA cloze sets. Each item is a narrative passage whose last word is withheld; the model must assign higher likelihood to that word than to alternatives, using discourse beyond the final sentence. Four configs are machine translations of the English OpenAI test split into German, Spanish, French and Italian. The fifth, English, loads Hugging Face config `en` rather than the `default` config used by lambada_openai. Text only.\n",
  "metric": {
   "baseline_note": "YAML metric_list records acc (mean, higher better) and perplexity (lower better). No random or human baseline is defined for open-vocabulary next-word prediction. The original 2016 LAMBADA paper's human filter applies to English source passages, not to these translations.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (next-word exact match); perplexity also reported",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LAMBADA multilingual (OpenAI MT)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1606.06031",
   "title": "The LAMBADA dataset: Word prediction requiring a broad discourse context",
   "url": "https://arxiv.org/abs/1606.06031",
   "year": 2016
  },
  "publisher": {
   "authors": [
    "Sid Black",
    "Denis Paperno",
    "Germ\u00e1n Kruszewski",
    "Angeliki Lazaridou",
    "Quan Ngoc Pham",
    "Raffaella Bernardi",
    "Sandro Pezzelle",
    "Marco Baroni",
    "Gemma Boleda",
    "Raquel Fern\u00e1ndez"
   ],
   "org": "EleutherAI (harness group and Hugging Face mirror); original LAMBADA, University of Trento CIMeC and University of Amsterdam",
   "url": "https://huggingface.co/datasets/EleutherAI/lambada_openai"
  },
  "released": "2023-07",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/lambada_multilingual",
  "saturation": {
   "as_of": "",
   "note": "No dated numeric leaderboard for the five mt tasks was opened here. English LAMBADA was already near ceiling in GPT-3's 2020 paper (see lambada.md). Translation quality, not discourse tracking, can dominate the non-English numbers.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness lambada_multilingual README (group and mt task list)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada_multilingual/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lambada_openai_mt_en YAML (group tag, EleutherAI/lambada_openai, acc and perplexity)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada_multilingual/lambada_mt_en.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lambada_openai_mt_de YAML (include en; task lambada_openai_mt_de; dataset_name de)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada_multilingual/lambada_mt_de.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lambada_openai YAML (dataset_name default; same scoring template)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada/lambada_openai.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/lambada_openai dataset card (cardData license mit; README Licensing section Modified MIT; 5,153 test rows per en/de/es/fr/it; googletrans script)",
    "url": "https://huggingface.co/datasets/EleutherAI/lambada_openai"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenAI GPT-2 Modified MIT licence (target of the EleutherAI card Licensing link)",
    "url": "https://raw.githubusercontent.com/openai/gpt-2/master/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API (created 2022-12-16; lastModified 2025-07-10; split counts)",
    "url": "https://huggingface.co/api/datasets/EleutherAI/lambada_openai"
   },
   {
    "accessed": "2026-09-08",
    "title": "translation_script.txt (Sid Black; googletrans en\u2192de/fr/it and related dest)",
    "url": "https://huggingface.co/datasets/EleutherAI/lambada_openai/raw/main/translation_script.txt"
   },
   {
    "accessed": "2026-09-08",
    "title": "Original LAMBADA paper (Paperno et al., 2016)",
    "url": "https://arxiv.org/abs/1606.06031"
   },
   {
    "accessed": "2026-09-08",
    "title": "Harness commits (add lambada_mt tasks 2023-07-05)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/commits?path=lm_eval/tasks/lambada_multilingual"
   }
  ],
  "status": "active",
  "subcategory": "multilingual last-word prediction on machine-translated LAMBADA",
  "summary": "lm-eval group of OpenAI-format LAMBADA last-word tests in English and machine-translated German, Spanish, French and Italian (5,153 passages each).",
  "tags": [
   "language-modelling",
   "cloze",
   "multilingual",
   "machine-translation",
   "lambada",
   "lm-eval"
  ],
  "task_format": "Causal language-model cloze scored as loglikelihood, not free generation. Prompt is the passage minus the last whitespace token; the target is a leading space plus that token. Runnable task names are lambada_openai_mt_{en,de,es,fr,it}.\n"
 }
}