{
 "body": "\n## What it measures\n\nThis group asks a language model to predict the last word of a short narrative\npassage. The English items are the OpenAI reformat of [LAMBADA](lambada.md):\nhumans could guess the word from the full passage, not from the last sentence\nalone. Non-English configs are new machine translations made for the Stable LM\n2 1.6B technical report. The authors inspected [lambada_multilingual](lambada_multilingual.md)\n(`EleutherAI/lambada_openai`) and called those googletrans files too noisy.\nThe intended skill is still discourse tracking. A bad translation of the target\nword can fail the item even when the model followed the story.\n\n## How it is scored\n\nEach YAML task uses loglikelihood scoring. Accuracy is the fraction of passages\nwhere the next-token ranking matches the gold last word. Perplexity is also\nreported and is lower-is-better. There is no answer list and no partial credit.\nThe group name `lambada_multilingual_stablelm` runs every `lambada_openai_mt_stablelm_*`\ntask that carries that tag. Do not mix these numbers with `lambada_openai`,\n`lambada_standard`, or `lambada_openai_mt_*` from the older multilingual group.\n\n## Dataset and licence\n\nThe harness loads `EleutherAI/lambada_multilingual_stablelm`. The Hub API lists\ntest-only configs en, de, es, fr, it, nl, pt, and default, each 5,153 rows.\nThe EleutherAI card has no licence field. The report points to\n`marcob/lambada_multilingual`, created 2024-01-24, card licence MIT, languages\nde/en/es/fr/it in YAML plus `lambada_test_nl.jsonl` and `lambada_test_pt_br.jsonl`\nin the repo. The harness README names five languages; nl and pt are extra\nYAML includes. Original English LAMBADA is CC BY 4.0 on `cimec/lambada`. Labels\nare the last tokens of public passages.\n\n## Who publishes it\n\nStability AI described the translations in the Stable LM 2 1.6B report\n(arXiv 2402.17834, 2024). EleutherAI hosts the harness group and a 2025-11-19\nHub mirror. The underlying cloze task is Paperno et al. 2016. There is no\nseparate live leaderboard for this group.\n\n## Lineage\n\nPredecessor: [lambada](lambada.md). Sibling group:\n[lambada_multilingual](lambada_multilingual.md), which still uses\n`EleutherAI/lambada_openai`. [lambada_cloze](lambada_cloze.md) is another\nEnglish formatting. The Stable LM files are a translation variant, not an\nalias of the older mt tasks.\n\n## Saturation and contamination\n\nEnglish LAMBADA has been easy for large models since GPT-3. Whether these\nretranslations still separate models was not established from a dated table\nhere. The passages are public, and the YAML turns decontamination on using\nthe full text as the query.\n\n## How to run it\n\nIn lm-evaluation-harness, run `lambada_multilingual_stablelm` or one\n`lambada_openai_mt_stablelm_{en,de,es,fr,it,nl,pt}` task. Confirm\n`dataset_path: EleutherAI/lambada_multilingual_stablelm`. Tokenization of the\ngold last word still drives accuracy, as on English LAMBADA. A score on\n`lambada_openai_mt_de` is not this German config.\n\n## Reading the numbers\n\nA high accuracy means the model assigned the gold last word the best\nlikelihood in that language\u2019s translation. It does not mean the model can\nwrite in that language. Compare language by language. Treat the README\u2019s\nfive-language list as incomplete relative to the YAML. If a paper cites\n`marcob/lambada_multilingual`, check whether it used `pt_br` rather than\n`pt`.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lambada_multilingual_stablelm",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "lambada_openai_mt_stablelm",
   "lambada_mt_stablelm"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "YAML sets should_decontaminate: true with the full passage as the query. English OpenAI LAMBADA has been public for years. The Stability AI files have been on Hugging Face since 2024-01-24 (marcob) and 2025-11-19 (EleutherAI mirror). Machine translation does not hide the items.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en",
    "de",
    "es",
    "fr",
    "it",
    "nl",
    "pt"
   ],
   "license": "MIT on marcob/lambada_multilingual; EleutherAI mirror card states no licence field",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 36071,
   "size_note": "EleutherAI/lambada_multilingual_stablelm has seven language configs plus default, each with a test split of 5,153 rows (en, de, es, fr, it, nl, pt). Running the full group scores 7 \u00d7 5,153 = 36,071 passages. The README lists only en, fr, de, it, es; nl and pt YAML files exist and the Hub has those configs. marcob/lambada_multilingual is the Stability AI upload cited in the paper (created 2024-01-24; MIT); it also has nl and pt_br files. This session did not byte-compare marcob pt_br with EleutherAI pt.\n",
   "splits": "test only, 5,153 rows per language config",
   "url": "https://huggingface.co/datasets/EleutherAI/lambada_multilingual_stablelm"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-053 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-053"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "lambada_multilingual_stablelm",
   "opencompass": "",
   "other": "Runnable tasks lambada_openai_mt_stablelm_{en,de,es,fr,it,nl,pt}. Dataset path EleutherAI/lambada_multilingual_stablelm, not EleutherAI/lambada_openai."
  },
  "id": "lambada_multilingual_stablelm",
  "last_updated": "2025-11",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "lambada",
   "successors": [],
   "variants": [
    "lambada_multilingual",
    "lambada_cloze"
   ]
  },
  "measures": "lambada_multilingual_stablelm is EleutherAI's lm-evaluation-harness group over machine-translated OpenAI-format LAMBADA cloze sets released for the Stable LM 2 1.6B report. Each item is a narrative passage whose last word is withheld. The model must rank that word above alternatives using discourse beyond the last sentence. Stability AI judged EleutherAI/lambada_openai translations too noisy and published new ones. Text only. Distinct from [lambada_multilingual](lambada_multilingual.md).\n",
  "metric": {
   "baseline_note": "YAML metric_list records acc (mean, higher better) and perplexity (lower better). No random or human baseline is defined for open-vocabulary next-word prediction. The 2016 LAMBADA human filter applies to English source passages, not these translations.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (next-word exact match); perplexity also reported",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "LAMBADA multilingual (Stable LM translations)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2402.17834",
   "title": "Stable LM 2 1.6B Technical Report",
   "url": "https://arxiv.org/abs/2402.17834",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Marco Bellagente",
    "Jonathan Tow",
    "Dakota Mahan"
   ],
   "org": "Stability AI (translations and report); EleutherAI (harness group and Hub mirror); original LAMBADA, University of Trento CIMeC and University of Amsterdam",
   "url": "https://huggingface.co/datasets/EleutherAI/lambada_multilingual_stablelm"
  },
  "released": "2024-02",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/lambada_multilingual_stablelm",
  "saturation": {
   "as_of": "",
   "note": "English LAMBADA was already near ceiling in GPT-3 (see lambada.md). No dated numeric table for these Stable LM translation tasks was copied out of the report HTML here. Translation quality can dominate non-English accuracy.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-eval lambada_multilingual_stablelm README (group, Stable LM report, task names)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada_multilingual_stablelm/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lambada_openai_mt_stablelm_en YAML (EleutherAI/lambada_multilingual_stablelm; acc and perplexity)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada_multilingual_stablelm/lambada_mt_stablelm_en.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lambada_openai_mt_stablelm_nl YAML (include en; dataset_name nl)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/lambada_multilingual_stablelm/lambada_mt_stablelm_nl.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "EleutherAI/lambada_multilingual_stablelm card (7 configs \u00d7 5,153 test; created 2025-11-19)",
    "url": "https://huggingface.co/datasets/EleutherAI/lambada_multilingual_stablelm"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API for the EleutherAI mirror (no license field)",
    "url": "https://huggingface.co/api/datasets/EleutherAI/lambada_multilingual_stablelm"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server split counts (5,153 test per config)",
    "url": "https://datasets-server.huggingface.co/info?dataset=EleutherAI/lambada_multilingual_stablelm"
   },
   {
    "accessed": "2026-09-08",
    "title": "marcob/lambada_multilingual (Stability AI upload; MIT; de/en/es/fr/it plus nl and pt_br files)",
    "url": "https://huggingface.co/datasets/marcob/lambada_multilingual"
   },
   {
    "accessed": "2026-09-08",
    "title": "marcob dataset API (created 2024-01-24; license mit)",
    "url": "https://huggingface.co/api/datasets/marcob/lambada_multilingual"
   },
   {
    "accessed": "2026-09-08",
    "title": "Stable LM 2 1.6B report HTML (rejects EleutherAI/lambada_openai MT; points to marcob)",
    "url": "https://ar5iv.labs.arxiv.org/html/2402.17834"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv abs 2402.17834",
    "url": "https://arxiv.org/abs/2402.17834"
   },
   {
    "accessed": "2026-09-08",
    "title": "Original LAMBADA paper (Paperno et al., 2016)",
    "url": "https://arxiv.org/abs/1606.06031"
   }
  ],
  "status": "active",
  "subcategory": "multilingual last-word prediction on Stable LM retranslations of OpenAI LAMBADA",
  "summary": "lm-eval group of OpenAI-format LAMBADA last-word tests using Stability AI's retranslations, not the older googletrans EleutherAI/lambada_openai files.",
  "tags": [
   "language-modelling",
   "cloze",
   "multilingual",
   "machine-translation",
   "lambada",
   "lm-eval"
  ],
  "task_format": "Causal LM cloze scored as loglikelihood. Prompt is the passage minus the last whitespace token; the target is a leading space plus that token. Group tag lambada_multilingual_stablelm. Runnable names lambada_openai_mt_stablelm_{en,de,es,fr,it,nl,pt}.\n"
 }
}