50,000 polarised IMDb movie reviews for binary sentiment classification, from a 2011 paper; frontier models sit far past the ceiling, so it now serves mainly as a robustness and calibration check.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | binary sentiment classification of movie reviews (legacy dataset, now mainly a calibration and robustness check) |
| Page status | saturated |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 50000 |
| Dataset licence | Not formally stated by the original release; the Hugging Face mirror's card lists the licence only as "other," without further detail, and the dataset's own official page states no explicit licence terms beyond a request to cite the paper. |
| Publisher | Stanford University |
IMDb asks a model to read a full movie review scraped from the IMDb website and classify it as positive or negative. Only strongly polarised reviews are included -- a rating of 7/10 or higher counts as positive, 4/10 or lower as negative, and the ambiguous middle range was deliberately excluded -- so the classification task itself is unambiguous even though real review text is often long, informal and mixed in tone. This predates the instruction-tuned LLM era entirely: it was built as a representation-learning benchmark for word-vector and embedding methods, not to probe reasoning, knowledge or instruction-following, and it is best read today as exactly that -- an old, well-understood classification dataset now used mainly to sanity-check a model's basic text-classification competence and robustness rather than to differentiate strong modern models.
Binary sentiment classification given the full text of one review; the model outputs "Positive" or "Negative", typically zero- or few-shot, scored by exact or quasi-exact match against the gold label.
No model card in ModelSpec reports this benchmark yet.