{
 "body": "\n## What it measures\n\nStory Cloze Test gives a model a four-sentence everyday story and two possible one-sentence endings, and asks it to pick the one that fits -- a framework the original authors describe as evaluating \"story understanding, story generation, and script learning,\" meant to replace an earlier \"Narrative Cloze Test.\" The instances are a small, separately-curated slice of a larger corpus the same paper introduced, ROCStories: roughly 50,000 (later expanded toward 98,000) five-sentence commonsense stories written by crowdworkers, intended mainly for training and story-generation research. From that pool, held-out four-sentence contexts were paired with one coherent and one incoherent candidate fifth sentence to build the evaluation set. A model with genuine narrative structure should prefer the coherent ending; the task was designed to be easy for a human and hard for shallow pattern-matching -- though, as described below, that goal did not fully hold up.\n\n## How it is scored\n\nAccuracy is the fraction of two-way choices answered correctly, an exact 50% random baseline. The companion LSDSem'17 shared-task paper frames success explicitly as \"getting closer to human performance of 100%.\" The original 2016 paper gives no single numeric baseline in its abstract, stating only that contemporary systems \"based on shallow language understanding struggle to achieve a high score\" -- well below that ceiling at release. Most current harnesses implement the choice via loglikelihood comparison (which candidate ending the model assigns higher probability) rather than free generation.\n\n## Dataset and licence\n\nSizes differ by edition, which matters more here than for most benchmarks. The commonly used, ungated Hugging Face mirror of the original data reports a \"2016\" configuration with 1,871 validation and 1,871 test instances (3,742 total, both public with answers), and a separate \"2018\" configuration with 1,571 validation instances and no released test split. The official Rochester project page separately states its latest combined release totals \"98,159 ROCStories and 3,744 Story Cloze Test instances\" -- a slightly different total from the per-edition figures above, reported as-is since no source ties them together exactly. Access to the official data requires a short free request form; no formal licence text was found on either the official site or the Hugging Face card, which tags the licence unknown.\n\n## Who publishes it\n\nStory Cloze Test and ROCStories come from Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli and James Allen, University of Rochester leading, published at NAACL HLT 2016. The Rochester group ran it as the LSDSem'17 shared task the following year (eight teams; a University of Washington team won), and the same core team published a 2018 follow-up addressing dataset bias, described below.\n\n## Lineage\n\nThe 2018 re-crowdsourced edition is best understood as a direct, bias-corrected revision of the same benchmark rather than a separate successor id, so this page discusses both together. A genuinely separate extension exists: XStoryCloze, a multilingual machine-translated version covering ten more languages, tagged on Hugging Face as extending arXiv:2112.10668; it has no page here, but OpenCompass uses its English configuration as a drop-in replacement (below). ROCStories, the larger source corpus, is a distinct artifact used mainly for story-generation research and also has no separate page. This repository's ANLI page (`anli`) notes ROCStories/StoryCloze material was used as one source domain for a later round of that benchmark's own passages.\n\n## Saturation and contamination\n\nNo current, working public leaderboard with a verifiable top score was found. The case for saturation instead rests on documented evidence: the benchmark's framing targets a 100% human ceiling on a binary choice, and a 2018 follow-up paper by three of the original authors, \"Tackling the Story Ending Biases in The Story Cloze Test,\" found some models beat early baselines by exploiting human-authorship artifacts in how right and wrong endings tended to be written -- sentence length, sentiment, word choice -- rather than genuine understanding. That is precisely why the re-crowdsourced 2018 edition exists, and the same paper reports the top model on the original dataset failed to hold its performance on the de-biased replacement -- direct evidence that high original-set scores partly measured an exploitable artifact. Contamination risk is high: instances have been publicly distributed and heavily mirrored for close to a decade, the set is small enough to memorise in full, and specific items are quoted verbatim across the paper, project site and countless downstream discussions.\n\n## How to run it\n\nlm-evaluation-harness's `storycloze` group runs both `storycloze_2016` and `storycloze_2018` as loglikelihood-based two-way multiple choice, loading the ungated `LSDSem/story_cloze` mirror directly with a decontamination check. OpenCompass takes a different approach: its own source comments state plainly that \"the original story cloze dataset and repo are not long[er] maintaining,\" and it evaluates the English configuration of XStoryCloze instead, via perplexity-based ranking. No HELM, inspect_evals or BIG-bench implementation was found. Because the two harnesses are not running the same underlying items -- original English StoryCloze versus a machine-translated extension's English slice -- their scores are not guaranteed comparable even when both are labelled \"Story Cloze accuracy.\"\n\n## Reading the numbers\n\nGiven the documented bias-exploitation history of the 2016 edition, a high score there is only weak evidence of genuine commonsense understanding -- it is also consistent with picking up on superficial stylistic differences between the two endings. Prefer a 2018-edition score where available, since that dataset was built specifically to remove the known shortcut, and check whether a number came from the original English mirror or OpenCompass's XStoryCloze substitute before comparing two scores. With only a two-way choice, an explicit 100% human ceiling, and the dataset's age, small size and heavy exposure, this benchmark reads today as a basic sanity check on narrative coherence rather than a way to meaningfully differentiate capable modern models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "storycloze",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "StoryCloze",
   "ROCStories Cloze Test"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The 2016 evaluation instances have been distributed (after a free registration step) since 2016, and are additionally mirrored, ungated, on Hugging Face; both the paper text and thousands of downstream papers, blog posts and benchmark-harness repositories discussing \"Story Cloze\" reproduce example stories and endings verbatim, including the specific right/wrong pairs. Combined with its small size (under 2,000 test instances per edition) and near-decade of public availability, this makes it one of the more straightforward benchmarks in this batch for a model to have memorised outright, on top of the separate, documented shortcut-exploitation issue described in Saturation and contamination above.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Not established from any source read for this page (Hugging Face tags it 'unknown'); the official Rochester site distributes the data free of charge after a mandatory access-request form",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 3742,
   "size_note": "Sizes differ meaningfully by edition, which matters for this dataset more than most: the widely used Hugging Face mirror of the original release (LSDSem/story_cloze) reports a \"2016\" (Spring 2016 / \"Winter 2016\" test-set) configuration of 1,871 validation and 1,871 test instances (3,742 total), and a separate \"2018\" configuration of 1,571 validation instances with no released test split, confirmed directly from that dataset's own card. The official Rochester project page separately states its latest combined release includes \"98,159 ROCStories and 3,744 Story Cloze Test instances\" -- a slightly different total from the per-edition Hugging Face figures above, which this page reports rather than reconciling, since no single source available for this research ties the two together exactly. The much larger ~50,000-98,000-story ROCStories corpus is a separate artifact from the cloze-test evaluation instances themselves and is not counted here.\n",
   "splits": "2016 edition: validation 1,871 / test 1,871 (both with public answers); 2018 edition: validation 1,571 only, no released test split",
   "url": "https://huggingface.co/datasets/LSDSem/story_cloze"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "storycloze_2016, storycloze_2018 (group: storycloze)",
   "opencompass": "story_cloze",
   "other": "lm-evaluation-harness's `storycloze` group runs both `storycloze_2016` and `storycloze_2018` as loglikelihood-based two-way multiple choice (which candidate ending is more probable given the four-sentence context), loading the ungated `LSDSem/story_cloze` mirror directly, with a decontamination check enabled. OpenCompass, by contrast, does not use the original English dataset at all: its own config comments state plainly that \"the original story cloze dataset and repo are not long[er] maintaining,\" and it instead evaluates the English configuration of XStoryCloze (`opencompass/xstory_cloze`) -- a multilingual, machine-translated extension of the original test set into ten additional languages, tagged on Hugging Face as extending arXiv:2112.10668 -- using perplexity-based ranking over the two endings. No HELM, inspect_evals or BIG-bench implementation was found.\n"
  },
  "id": "storycloze",
  "last_updated": "2018-07",
  "leaderboard_url": "https://competitions.codalab.org/competitions/15333",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Story Cloze Test gives a model a four-sentence everyday story and two possible one-sentence endings, and asks it to pick the ending that actually fits -- a \"commonsense reasoning framework for evaluating story understanding, story generation, and script learning,\" in the original authors' own words, designed to replace the earlier \"Narrative Cloze Test\" as a way to measure whether a system has learned the causal and temporal structure of ordinary events. The test set is a small, separately-curated slice of a much larger corpus the same paper introduced, ROCStories: roughly 50,000 five-sentence commonsense stories written by crowdworkers, intended for training and for story-generation research, from which the smaller cloze-test instances (four-sentence context plus right/wrong endings) were built as a held-out evaluation set. A model that has learned genuine narrative and commonsense structure should prefer the coherent ending; the task is designed to be easy for a human reader and hard for a system relying only on shallow textual cues.\n",
  "metric": {
   "baseline_note": "The companion LSDSem'17 shared-task paper frames the goal explicitly as \"getting closer to human performance of 100%,\" treating human accuracy on this forced choice as effectively perfect. The original 2016 paper does not give a single specific baseline accuracy figure in its abstract, stating only that contemporary \"state-of-the-art models based on shallow language understanding struggle to achieve a high score\" -- i.e. well below the human ceiling at release. Random baseline is exactly 50% given a binary choice.\n",
   "direction": "higher_is_better",
   "human_baseline": 100,
   "max_score": 100,
   "name": "Accuracy (2-way forced choice)",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "Story Cloze Test",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1604.01696",
   "title": "A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories",
   "url": "https://arxiv.org/abs/1604.01696",
   "year": 2016
  },
  "publisher": {
   "authors": [
    "Nasrin Mostafazadeh",
    "Nathanael Chambers",
    "Xiaodong He",
    "Devi Parikh",
    "Dhruv Batra",
    "Lucy Vanderwende",
    "Pushmeet Kohli",
    "James Allen"
   ],
   "org": "University of Rochester",
   "url": "https://cs.rochester.edu/nlp/rocstories/"
  },
  "released": "2016-04",
  "repo_url": "https://cs.rochester.edu/nlp/rocstories/",
  "saturation": {
   "as_of": "",
   "note": "No current, working public leaderboard with a verifiable top score was found for this research (the original CodaLab competition page still resolves but does not render a current results table through the tooling used here). The case for treating this benchmark as saturated instead rests on documented evidence rather than a single top-score figure: the shared-task framing explicitly targets a 100% human ceiling on a binary choice, and a 2018 follow-up paper by three of the original authors, \"Tackling the Story Ending Biases in The Story Cloze Test\" (Sharma, Allen, Bakhshandeh and Mostafazadeh, ACL 2018), found that some models were beating early baselines by exploiting human-authorship artifacts in how right and wrong endings were written -- stylistic tells such as sentence length, sentiment or specific word choices that differ systematically between correct and incorrect endings, independent of story comprehension -- rather than through genuine narrative understanding. That finding is why a second, re-crowdsourced \"2018\" edition exists at all, and the same paper reports that the top-performing model on the original dataset failed to hold its performance on the de-biased replacement. No model card in this repository currently reports this benchmark (checked by grep across models/).\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Stories (arXiv abstract: authors, Story Cloze Test definition, NAACL HLT 2016)",
    "url": "https://arxiv.org/abs/1604.01696"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official ROCStories / Story Cloze Test project page (rendered): access-request process, combined 98,159/3,744-instance release figures, example items",
    "url": "https://cs.rochester.edu/nlp/rocstories/"
   },
   {
    "accessed": "2026-09-08",
    "title": "LSDSem'17 Shared Task page (rendered): 8 teams competed, UW team won the first challenge, links to the Winter 2016 paper and CodaLab competition",
    "url": "https://www.cs.rochester.edu/nlp/rocstories/LSDSem17/"
   },
   {
    "accessed": "2026-09-08",
    "title": "LSDSem 2017 Shared Task: The Story Cloze Test (Mostafazadeh et al., 2017) -- states the explicit 100% human-performance framing",
    "url": "https://aclanthology.org/W17-0906/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Tackling the Story Ending Biases in The Story Cloze Test (Sharma, Allen, Bakhshandeh and Mostafazadeh, ACL 2018) -- documents the annotation-artifact exploitation and the resulting de-biased 2018 dataset",
    "url": "https://aclanthology.org/P18-2119/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official CodaLab Story Cloze Test competition page (rendered): confirms the Winter 2016 bias finding and the recommendation to use Winter 2018 instead",
    "url": "https://competitions.codalab.org/competitions/15333"
   },
   {
    "accessed": "2026-09-08",
    "title": "LSDSem/story_cloze dataset card (ungated mirror): 2016 and 2018 config feature schema and exact split sizes",
    "url": "https://huggingface.co/datasets/LSDSem/story_cloze/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "juletxara/xstory_cloze dataset API record (confirms arxiv:2112.10668 tag and source_datasets:extended|story_cloze, the multilingual variant OpenCompass uses)",
    "url": "https://huggingface.co/api/datasets/juletxara/xstory_cloze"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness storycloze_2016.yaml (LSDSem/story_cloze mirror; loglikelihood multiple_choice; decontamination enabled)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/storycloze/storycloze_2016.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass storycloze_ppl_496661.py config (explicit comment on the original repo no longer being maintained; switches to opencompass/xstory_cloze)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/storycloze/storycloze_ppl_496661.py"
   }
  ],
  "status": "saturated",
  "subcategory": "commonsense story-ending selection (binary forced-choice cloze test)",
  "summary": "Story Cloze Test asks a model to pick the correct one of two endings to a four-sentence story; the original 2016 set has documented annotation biases exploitable without real story understanding.",
  "tags": [
   "commonsense-reasoning",
   "cloze-test",
   "story-understanding",
   "saturated",
   "forced-choice"
  ],
  "task_format": "Two-way forced choice: given a four-sentence story and two candidate fifth sentences, select the one that is the coherent, correct ending. Most harnesses implement this as loglikelihood comparison (which candidate ending is more probable given the context) rather than free generation.\n"
 }
}