lm-eval group of OpenAI-format LAMBADA last-word tests in English and machine-translated German, Spanish, French and Italian (5,153 passages each).
unassessed
| Category | reasoning |
|---|---|
| Subcategory | multilingual last-word prediction on machine-translated LAMBADA |
| Page status | active |
| Metric | accuracy (next-word exact match); perplexity also reported |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 25765 |
| Dataset licence | MIT (Hugging Face cardData); README Licensing section cites Modified MIT (OpenAI GPT-2) |
| Publisher | EleutherAI (harness group and Hugging Face mirror); original LAMBADA, University of Trento CIMeC and University of Amsterdam |
lambada_multilingual is EleutherAI's lm-evaluation-harness group over five OpenAI-format LAMBADA cloze sets. Each item is a narrative passage whose last word is withheld; the model must assign higher likelihood to that word than to alternatives, using discourse beyond the final sentence. Four configs are machine translations of the English OpenAI test split into German, Spanish, French and Italian. The fifth, English, loads Hugging Face config `en` rather than the `default` config used by lambada_openai. Text only.
Causal language-model cloze scored as loglikelihood, not free generation. Prompt is the passage minus the last whitespace token; the target is a leading space plus that token. Runnable task names are lambada_openai_mt_{en,de,es,fr,it}.
No model card in ModelSpec reports this benchmark yet.