{
 "body": "\n## What it measures\n\nSynthetic Reasoning (Natural Language) is a HELM scenario that tests rule-based deduction phrased\nas simple English sentences: the model is given conditional rules of the form \"things that are\n[attribute] are also [attribute]\" together with facts about specific subjects, and must produce the\nset of additional facts that logically follow. The scenario's own docstring describes it as\ninspired by \"Transformers as Soft Reasoners over Language\" (Clark, Tafjord and Richardson,\narXiv:2002.05867), and HELM's scenario metadata additionally frames it as based on LIME (Wu et al.,\n2021, PMLR). It shares its underlying rule-and-fact generation logic with HELM's companion\n`synthetic_reasoning` scenario, which states the same kind of problem in an abstract symbolic\nnotation instead of natural-language sentences \u2014 so the pairing is designed to isolate how much a\nmodel's reasoning depends on symbolic versus linguistic framing.\n\nDifficulty is controlled by how abstract the subjects and attributes are (concrete, familiar names\nversus more abstracted placeholders), which lets the scenario probe whether a model is doing genuine\nrule-chaining or leaning on surface familiarity with common English predicates.\n\n## How it is scored\n\nThe model generates free-form text listing the facts it deduces; HELM scores this scenario using a\nfamily of set-comparison metrics \u2014 `f1_set_match`, `iou_set_match` and `exact_set_match` \u2014 comparing\nthe model's output set against the gold consequent set, with `f1_set_match` treated as the primary\nmetric. Evaluation is generation-based (not multiple choice), and HELM's classic run spec prompts\nthe model with a generic \"solve the following problem\" instruction rather than a task-specific\nrubric.\n\n## Dataset and licence\n\nUnlike most benchmarks in this repository, this scenario has no fixed downloadable dataset file:\nits `get_instances` method procedurally generates 1,000 training, 5,000 validation and 5,000 test\ninstances per difficulty setting from a built-in vocabulary of subjects, attributes and rule\ntemplates defined directly in the HELM source code. There is therefore no separate dataset licence\nto report beyond HELM's own repository licence, and no fixed \"held-out answer\" file exists outside\nthe generation code itself, which is public.\n\n## Who publishes it\n\nSynthetic Reasoning (Natural Language) is maintained by Stanford's Center for Research on\nFoundation Models (CRFM) as part of HELM, introduced in \"Holistic Evaluation of Language Models\"\n(Liang et al., arXiv:2211.09110, 2022). The scenario's own credited inspirations are Clark, Tafjord\nand Richardson's RuleTaker work and Wu et al.'s LIME paper, rather than a dedicated paper for this\nscenario itself.\n\n## Lineage\n\nThis scenario is the natural-language counterpart to HELM's `synthetic_reasoning` scenario, which\nposes the same style of rule-and-fact deduction problem using an abstract symbolic notation instead\nof English sentences; the two are best read as paired variants rather than one superseding the\nother. No predecessor or successor scenario under this exact id exists in HELM or in this\nrepository.\n\n## Saturation and contamination\n\nThis page did not independently open a current HELM leaderboard table, so saturation status for\nthis scenario is unknown. Contamination risk is judged low because items are generated\nprocedurally at evaluation time from a fixed template and vocabulary rather than drawn from a\nstatic published dataset, making verbatim memorization of specific test instances unlikely, though\nthe generation templates and vocabulary are themselves visible in HELM's public source code.\n\n## How to run it\n\nRun through the HELM framework using the `synthetic_reasoning_natural` run spec\n(`helm.benchmark.scenarios.synthetic_reasoning_natural_scenario.SRNScenario`), which accepts a\n`difficulty` parameter selecting among the scenario's abstraction levels. Because the dataset is\ngenerated rather than fixed, exact reported numbers can depend on the HELM version's generation\ncode and on which difficulty setting and instance counts a given run configuration used \u2014 check\nboth before comparing scores across reports.\n\n## Reading the numbers\n\nA high `f1_set_match` score indicates a model can chain simple conditional rules stated in plain\nEnglish and correctly enumerate their logical consequences, without needing an abstract symbolic\nrepresentation of the problem. It does not, on its own, establish performance on the harder\nsymbolic `synthetic_reasoning` variant, on naturalistic multi-hop reasoning over real-world facts,\nor on rule sets larger or more complex than this scenario's fixed generation templates produce.\nBecause no live leaderboard was checked for this page, treat any specific score as needing\nverification against a current HELM run before drawing conclusions about how strong it is.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "synthetic_reasoning_natural",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SRN"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Items are procedurally generated from a fixed template and vocabulary at evaluation time rather than drawn from a static, previously published corpus, so a specific test instance is unlikely to have appeared verbatim in pretraining data, though the generation templates and vocabulary themselves are public in the HELM source code.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The scenario is procedurally generated at run time (not a fixed downloaded file): the scenario code builds 1,000 training, 5,000 validation and 5,000 test instances per difficulty setting from a fixed vocabulary of subjects, attributes and rule templates, rather than sampling from a static, citable corpus. Exact total item counts as reported on the HELM leaderboard depend on the eval configuration (e.g. how many test instances are actually scored) and were not independently confirmed from a leaderboard run for this page.\n",
   "splits": "train (1,000) / valid (5,000) / test (5,000), procedurally generated per difficulty level",
   "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/synthetic_reasoning_natural_scenario.py"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-007 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-007"
  },
  "harness": {
   "bigbench": "",
   "helm": "synthetic_reasoning_natural",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "synthetic_reasoning_natural",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "synthetic_reasoning"
   ]
  },
  "measures": "The model is given a small set of natural-language conditional rules (if a thing has certain attributes, it has certain other attributes) and a set of facts about specific subjects, then must deduce which consequent facts logically follow, generating its answer as text. Unlike the companion `synthetic_reasoning` scenario, which states rules and facts in an abstract symbolic notation, this variant wraps the same underlying deduction problem in simple natural-language sentences, testing whether a model can pattern-match and chain rules when they are phrased linguistically rather than symbolically. Three difficulty levels vary how abstract the subjects and attributes are (concrete named things vs. abstracted placeholders), probing whether performance depends on surface familiarity rather than the underlying logical structure.\n",
  "metric": {
   "baseline_note": "HELM's classic run spec scores this scenario with a set of basic metrics \u2014 f1_set_match, iou_set_match and exact_set_match \u2014 comparing the set of facts the model outputs to the gold consequent set; f1_set_match is treated as HELM's primary metric for this scenario. No random-guess or human baseline is published in the scenario code or HELM's scenario metadata.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "f1_set_match (also reports iou_set_match and exact_set_match)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Synthetic Reasoning (Natural Language)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2211.09110",
   "title": "Holistic Evaluation of Language Models",
   "url": "https://arxiv.org/abs/2211.09110",
   "year": 2022
  },
  "publisher": {
   "authors": [],
   "org": "Stanford CRFM (HELM)",
   "url": "https://crfm.stanford.edu/helm/"
  },
  "released": "2022",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/synthetic_reasoning_natural_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "This page did not independently open a live HELM leaderboard table for this scenario, so current top scores and how close leading models sit to the ceiling are not established here.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM: synthetic_reasoning_natural_scenario.py",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/synthetic_reasoning_natural_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM classic_run_specs.py (synthetic_reasoning_natural run spec)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Transformers as Soft Reasoners over Language",
    "url": "https://arxiv.org/abs/2002.05867"
   },
   {
    "accessed": "2026-09-08",
    "title": "LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning (Wu et al., 2021)",
    "url": "https://proceedings.mlr.press/v139/wu21c.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "Holistic Evaluation of Language Models",
    "url": "https://arxiv.org/abs/2211.09110"
   }
  ],
  "status": "active",
  "subcategory": "procedurally-generated rule-based deduction, stated in natural-language sentences",
  "summary": "HELM scenario: given natural-language conditional rules and facts, deduce the correct consequent, at easy/medium/hard abstraction levels.",
  "tags": [
   "synthetic",
   "rule-based-reasoning",
   "deduction",
   "natural-language",
   "helm"
  ],
  "task_format": "Free-form generation: model must output the set of facts that follow from the given rules; graded as a set-matching problem, not multiple choice."
 }
}