{
 "body": "\n## What it measures\n\nUnderstanding Fables presents a paraphrased fable and five candidate morals. The model selects the moral that best captures the story's lesson. Anthropomorphized animals and objects make the narrative domain unusual.\n\nIt combines reading comprehension, moral abstraction and distractor discrimination. It is not a test of knowledge of original Aesop wording.\n\n## How it is scored\n\nThe preferred metric is multiple_choice_grade. Five alternatives make 20% uniform chance. The README reports GPT2 at 0.19 and zero-shot BART-Large, GPT2-NEO-1.3B and RoBERTa-Large results under different scoring procedures.\n\nThose are historical results rather than a current leaderboard. Report model, task version and whether sentence or pseudo-perplexity ranks candidates.\n\n## Dataset and licence\n\nThe README states that 189 unique fables remain after duplicate and near-duplicate removal. Stories came from Aesop Fables pages and were paraphrased by a native-like English speaker with literature training. Entities, structure and register were changed while morals were minimally updated.\n\nA separate data licence is not established. A public task repository does not settle rights for source fables or derived text.\n\n## Who publishes it\n\nBIG-bench hosts the task and credits Denis Emelin. Its generated header reports 189 multiple-choice dummy-model queries. No current standalone leaderboard was established.\n\n## Lineage\n\nThe README names English Proverbs as related, but Understanding Fables uses longer stories and five-choice morals. No successor or predecessor page was established.\n\n## Saturation and contamination\n\nSource stories are public, creating leakage risk. Paraphrasing and changed entities reduce exact memorization but do not create a private test. Historical near-chance results do not establish current saturation.\n\n## How to run it\n\nRun BIG-bench task understanding_fables with five choices and four distractors. Preserve the paraphrased items; do not substitute original Aesop text.\n\n## Reading the numbers\n\nAbove 20% indicates some ability to select intended morals. It does not prove broad moral reasoning. Check errors involving idioms, anthropomorphic actors and longer narratives.\n\nThe task also rewards distinguishing a central lesson from a merely related phrase. Scores can therefore reflect familiarity with proverb-like language, narrative compression and the quality of the distractor construction, in addition to story comprehension.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "understanding_fables",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "BIG-bench Understanding Fables"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Source fables are public, but the task paraphrases stories and changes participants and wording.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 189,
   "size_note": "189 paraphrased, unique fables after duplicate and problematic-story filtering.",
   "splits": "single task set; no train/test split stated",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/understanding_fables"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-new-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-003"
  },
  "harness": {
   "bigbench": "understanding_fables",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "understanding_fables",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "big_bench",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The model reads a short fable and selects the moral that best expresses its lesson. Fables are paraphrased from Aesop-related sources with plausible distractor morals.",
  "metric": {
   "baseline_note": "Five alternatives imply 20% chance; GPT2 was reported at 0.19.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "multiple_choice_grade accuracy",
   "random_baseline": 20,
   "unit": "%"
  },
  "name": "Understanding Fables",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Denis Emelin"
   ],
   "org": "BIG-bench collaboration",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/understanding_fables"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "Historical results do not establish a current ceiling.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench Understanding Fables README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/understanding_fables/README.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "active",
  "subcategory": "narrative moral selection",
  "summary": "189 paraphrased fables paired with five candidate morals test narrative understanding and cross-domain generalization.",
  "tags": [
   "big-bench",
   "narrative",
   "reasoning",
   "multiple-choice"
  ],
  "task_format": "Five-choice multiple choice with one correct moral and four distractors."
 }
}