WikiText scores raw language-modelling quality by perplexity on curated Wikipedia articles; it measures how well a model predicts text, not whether it answers a task correctly.
unassessed
| Category | generation |
|---|---|
| Subcategory | language-modelling perplexity over curated Wikipedia text |
| Page status | active |
| Metric | perplexity (word-level and/or byte-level; bits-per-byte is also reported by some harnesses) |
| Direction | lower_is_better |
| Unit | perplexity |
| Dataset licence | CC BY-SA 3.0 and GFDL (dual-licensed, per the Hugging Face dataset card; inherited from Wikipedia's own article licensing) |
| Publisher | MetaMind, a Salesforce company (now Salesforce Research) |
WikiText is not a task benchmark: it gives a model a stream of curated Wikipedia article text and measures how well the model's predicted next-token probabilities match what actually comes next, scored as perplexity. There are no questions, answers or instructions to follow, so a WikiText score says nothing directly about a model's ability to solve a problem, follow an instruction or reason -- it measures modelling quality, the calibration and fluency of the model's underlying probability distribution over English text, which is a different and narrower thing than the accuracy percentage reported by a question-answering or reasoning benchmark.
A model reads long-form Wikipedia article text (WikiText-2 or the ~110x larger WikiText-103) and, for each token, produces a probability distribution over the vocabulary; no prompt, instruction or answer is generated. Evaluation is typically "rolling" loglikelihood scoring across each document, letting a model use as much preceding context as its window allows rather than scoring fixed, isolated chunks.
No model card in ModelSpec reports this benchmark yet.