Story Cloze Test asks a model to pick the correct one of two endings to a four-sentence story; the original 2016 set has documented annotation biases exploitable without real story understanding.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | commonsense story-ending selection (binary forced-choice cloze test) |
| Page status | saturated |
| Metric | Accuracy (2-way forced choice) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 3742 |
| Dataset licence | Not established from any source read for this page (Hugging Face tags it 'unknown'); the official Rochester site distributes the data free of charge after a mandatory access-request form |
| Publisher | University of Rochester |
Story Cloze Test gives a model a four-sentence everyday story and two possible one-sentence endings, and asks it to pick the ending that actually fits -- a "commonsense reasoning framework for evaluating story understanding, story generation, and script learning," in the original authors' own words, designed to replace the earlier "Narrative Cloze Test" as a way to measure whether a system has learned the causal and temporal structure of ordinary events. The test set is a small, separately-curated slice of a much larger corpus the same paper introduced, ROCStories: roughly 50,000 five-sentence commonsense stories written by crowdworkers, intended for training and for story-generation research, from which the smaller cloze-test instances (four-sentence context plus right/wrong endings) were built as a held-out evaluation set. A model that has learned genuine narrative and commonsense structure should prefer the coherent ending; the task is designed to be easy for a human reader and hard for a system relying only on shallow textual cues.
Two-way forced choice: given a four-sentence story and two candidate fifth sentences, select the one that is the coherent, correct ending. Most harnesses implement this as loglikelihood comparison (which candidate ending is more probable given the context) rather than free generation.
No model card in ModelSpec reports this benchmark yet.