{
 "body": "\n## What it measures\n\nlm-eval `moral_stories` is a two-way choice over English social actions. The model\nsees a short norm, a situation, and an intention. It must assign higher likelihood\nto the crowd-written moral action than to the matching immoral action. Both actions\nare written to satisfy the same intention. Only one of them follows the norm.\n\nThis is not a free-text ethics essay, and it is not BIG-bench\n[Moral Permissibility](moral_permissibility.md). The 2021 paper used the same\nstories for generation and several classification splits. The harness scores one\nfixed likelihood comparison on the full 12k set.\n\n## How it is scored\n\nThe YAML sets `output_type: multiple_choice` and reports mean `acc` and `acc_norm`.\n`utils.process_docs` always lists `[moral_action, immoral_action]` and `label: 0`.\nA correct item is one where the moral action has the higher (or length-normalised\nhigher) log-likelihood. Chance is 50%. No few-shot pool is declared. Do not treat\nthe paper's RoBERTa action-classification table as this score.\n\n## Dataset and licence\n\nEmelin et al. collected about 14k Mechanical Turk stories from Social-Chem-101\nnorms and kept 12k after validation. Each story has seven sentences: norm,\nsituation, intention, moral action, moral consequence, immoral action, immoral\nconsequence. The GitHub LICENSE and the `demelin/moral_stories` card are MIT.\nThe LabHC copy used by lm-eval is the same 12,000 rows and does not restate a\nlicence on its card. Answers are public.\n\n## Who publishes it\n\nThe dataset is Emelin, Le Bras, Hwang, Forbes and Choi (EMNLP 2021; arXiv\n2012.15738), with authors at Edinburgh, AI2 and the University of Washington.\nThe ranking task is maintained in EleutherAI lm-evaluation-harness. LabHC\nrepublished the full English subset next to Histoires Morales.\n\n## Lineage\n\nNo predecessor page. [Moral Permissibility](moral_permissibility.md) is a different\nBIG-bench trolley-style yes/no set. Histoires Morales is a French translation of\nthese 12k stories; it has no page here. The paper's generation and split-specific\nclassification tasks are not this harness id.\n\n## Saturation and contamination\n\nNo current top score for the likelihood task was read. Contamination is high:\npublic stories and both actions since 2021, and lm-eval evaluates the public\ntrain split.\n\n## How to run it\n\n`lm_eval --tasks moral_stories`. Confirm you are scoring likelihood, not a\ngenerated action. The Histoires Morales French dump is a different dataset path.\n\n## Reading the numbers\n\nA high score means the model prefers the labelled moral action under this\ncrowd-written US-centric code of conduct. It does not mean the model can state\na new norm, predict a consequence, or refuse harm in an open prompt. Compare\nonly to other `moral_stories` likelihood runs, and read the original paper when\nthe claim is generation quality.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "moral_stories",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MoralStories"
  ],
  "category": "safety",
  "contamination": {
   "note": "The full 12k stories have been public since 2021 (GitHub, TinyURL dump, Hugging Face). lm-eval scores the public train split. The LabHC copy was published to sit beside the French Histoires Morales translation of the same stories.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT (demelin/moral_stories card and GitHub LICENSE; LabHC card states no licence field)",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 12000,
   "size_note": "LabHC/moral_stories (the harness dataset_path) is the \"full\" 12,000-story subset with a single train split of 12,000 rows, confirmed from the Hugging Face dataset card and API. The original paper and demelin/moral_stories also state 12k structured narratives after validation of about 14k collected stories. lm-eval sets test_split to train, so all 12,000 stories are scored. The official Hugging Face dump also ships many classification and generation splits (norm-distance, lexical-bias, minimal-pairs) that this task does not use.\n",
   "splits": "LabHC dump: train only (12,000). lm-eval evaluates that train split.",
   "url": "https://huggingface.co/datasets/LabHC/moral_stories"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-013 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-013"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "moral_stories",
   "opencompass": "",
   "other": "Task YAML version 1.0. dataset_path LabHC/moral_stories. process_docs in lm_eval/tasks/moral_stories/utils.py. The harness README says the implementation follows the Histoires Morales work (LabHC/histoires_morales).\n"
  },
  "id": "moral_stories",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "This id is EleutherAI lm-evaluation-harness task moral_stories, not the paper's original generation suite and not BIG-bench moral_permissibility. Each item is an English seven-part story. The harness concatenates the norm, situation and intention, then asks which of two action sentences is more likely: the crowd-written moral action or the immoral action. The labelled target is always the moral action. The original EMNLP 2021 work instead asked models to generate actions, consequences or norms under those constraints.\n",
  "metric": {
   "baseline_note": "Two choices, so chance is 50%. The harness always places the moral action first and sets label 0, so the score is the share of items where the moral action has higher likelihood. The paper reports classification accuracy for several grounded action-classification settings on RoBERTa-large, not this likelihood ranking, so those figures are not copied in as this page's human or model baseline.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (acc); length-normalised accuracy (acc_norm)",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "Moral Stories",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2012.15738",
   "title": "Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences",
   "url": "https://aclanthology.org/2021.emnlp-main.54/",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Denis Emelin",
    "Ronan Le Bras",
    "Jena D. Hwang",
    "Maxwell Forbes",
    "Yejin Choi"
   ],
   "org": "Allen Institute for AI; University of Edinburgh; University of Washington",
   "url": "https://github.com/demelin/moral_stories"
  },
  "released": "2021",
  "repo_url": "https://github.com/demelin/moral_stories",
  "saturation": {
   "as_of": "",
   "note": "No current public LLM leaderboard for the lm-eval likelihood task was opened. The 2021 paper's RoBERTa action-classification numbers are a different protocol.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness moral_stories.yaml (task name, LabHC path, acc / acc_norm)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/moral_stories/moral_stories.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval moral_stories utils.process_docs (moral action always label 0)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/moral_stories/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval Moral Stories README (seven-part schema; Histoires Morales note)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/moral_stories/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "LabHC/moral_stories card (12,000 train rows; full subset)",
    "url": "https://huggingface.co/datasets/LabHC/moral_stories"
   },
   {
    "accessed": "2026-09-08",
    "title": "LabHC/moral_stories Hugging Face API (num_examples 12000)",
    "url": "https://huggingface.co/api/datasets/LabHC/moral_stories"
   },
   {
    "accessed": "2026-09-08",
    "title": "demelin/moral_stories card (MIT; original dump and splits)",
    "url": "https://huggingface.co/datasets/demelin/moral_stories"
   },
   {
    "accessed": "2026-09-08",
    "title": "demelin/moral_stories repository and MIT LICENSE",
    "url": "https://github.com/demelin/moral_stories"
   },
   {
    "accessed": "2026-09-08",
    "title": "EMNLP 2021 Moral Stories paper",
    "url": "https://aclanthology.org/2021.emnlp-main.54/"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv 2012.15738 HTML (12k stories; classification versus generation tasks)",
    "url": "https://ar5iv.labs.arxiv.org/html/2012.15738"
   }
  ],
  "status": "unknown",
  "subcategory": "binary ranking of a moral versus immoral action given a social norm and context",
  "summary": "lm-eval ranks a crowd-written moral action against an immoral one, given a social norm, situation and intention from the 12k-story Moral Stories corpus.",
  "tags": [
   "social-reasoning",
   "morality",
   "multiple-choice",
   "lm-eval",
   "english"
  ],
  "task_format": "Two-way multiple_choice over moral_action versus immoral_action. Context is the capitalised norm, situation and intention. Zero extra few-shot examples in the YAML. English text.\n"
 }
}