lm-eval group of OpenAI-format LAMBADA last-word tests using Stability AI's retranslations, not the older googletrans EleutherAI/lambada_openai files.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | multilingual last-word prediction on Stable LM retranslations of OpenAI LAMBADA |
| Page status | active |
| Metric | accuracy (next-word exact match); perplexity also reported |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 36071 |
| Dataset licence | MIT on marcob/lambada_multilingual; EleutherAI mirror card states no licence field |
| Publisher | Stability AI (translations and report); EleutherAI (harness group and Hub mirror); original LAMBADA, University of Trento CIMeC and University of Amsterdam |
lambada_multilingual_stablelm is EleutherAI's lm-evaluation-harness group over machine-translated OpenAI-format LAMBADA cloze sets released for the Stable LM 2 1.6B report. Each item is a narrative passage whose last word is withheld. The model must rank that word above alternatives using discourse beyond the last sentence. Stability AI judged EleutherAI/lambada_openai translations too noisy and published new ones. Text only. Distinct from [lambada_multilingual](lambada_multilingual.md).
Causal LM cloze scored as loglikelihood. Prompt is the passage minus the last whitespace token; the target is a leading space plus that token. Group tag lambada_multilingual_stablelm. Runnable names lambada_openai_mt_stablelm_{en,de,es,fr,it,nl,pt}.
No model card in ModelSpec reports this benchmark yet.