{
 "body": "## What it measures\n\nWorldSense is a synthetic benchmark designed to test whether a model can build and reliably reason over a small internal \"world model\" from a set of natural-language statements, while controlling for dataset biases that let models shortcut similar tasks without real understanding. The Inspect Evals implementation exposes a single `worldsense` task covering three problem types: Infer (judge whether a statement is true or false given the scenario), Compl (pick the one correct statement out of three options), and Consist (judge whether a set of statements is jointly possible or impossible). Each item also carries a difficulty grade, \"trivial\" (solvable from the statements alone) or \"normal\" (requiring the model to actually track world state).\n\n## How it is scored\n\nThe task is scored by accuracy on the underlying true/false or multiple-choice judgement, with standard error reported alongside. Because the benchmark is explicitly built to control for response-position and label-frequency biases, the harness also computes a weighted accuracy that adjusts for tuple ID, problem name and problem size, intended to be more robust to a model exploiting superficial answer patterns than raw accuracy alone.\n\n## Dataset and licence\n\nWorldSense's items are synthetically generated rather than drawn from an existing text corpus, built around templated scenarios such as object placement and scheduling puzzles. Neither the Inspect Evals README nor the paper abstract read for this page states a total item count or an explicit dataset licence; both are left unknown here rather than guessed.\n\n## Who publishes it\n\nWorldSense was introduced by Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Gr\u00e9goire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes and Pascal Vincent in \"WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models\" (arXiv:2311.15930, submitted November 2023). Author affiliation was not confirmed from the sources opened for this page and is left unstated. The UK Government's AI Security Institute (via the Inspect Evals project) maintains this harness integration; no separate public leaderboard was found.\n\n## Lineage\n\nNo predecessor or successor benchmark was identified for WorldSense in the sources read for this page. It is one of several synthetic, bias-controlled reasoning benchmarks that emerged around 2023-2024 in response to concerns that natural-corpus reasoning benchmarks can be solved via surface patterns rather than genuine world modelling; related benchmarks of that kind are not yet pages in this repository.\n\n## Saturation and contamination\n\nNo saturation status was established from the sources read for this page. The original paper reports that GPT-3.5, GPT-4 and Llama-2-chat all make errors even with as few as three objects in the scenario, exhibit response biases, and do not reliably improve with chain-of-thought prompting; fine-tuning improved scores but did not generalise beyond the trained problem space, per the abstract. Because items are synthetically generated rather than scraped from existing text, direct verbatim contamination is less likely than for corpus-derived benchmarks, but this was not independently studied in the sources read.\n\n## How to run it\n\nRun the `worldsense` task in the Inspect Evals framework (`UKGovernmentBEIS/inspect_evals`). Results can be filtered or aggregated by problem type (Infer, Compl, Consist), problem name and difficulty grade; report which subset and grade a score covers, since the \"trivial\" and \"normal\" grades are designed to have very different difficulty.\n\n## Reading the numbers\n\nA high WorldSense score, especially on the \"normal\" grade, indicates a model can track and reason over an explicit small world model rather than answering from surface statement patterns; a model that scores well only on \"trivial\" items has not demonstrated that. Because the benchmark's own weighted-accuracy metric exists specifically to correct for response-position and label biases, a raw accuracy figure without the weighted variant should be treated cautiously. The original paper's finding that even strong 2023-era chat models erred with as few as three tracked objects suggests this benchmark can surface reasoning limits that other, less bias-controlled tasks miss.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "worldsense",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "Items are synthetically generated, which limits exposure to pre-existing training corpora, but no contamination study was found.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Exact dataset size was not stated in the README or paper abstract read for this page.",
   "splits": "",
   "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/worldsense"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-001 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "worldsense",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "worldsense",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "WorldSense tests whether a model can maintain a consistent internal world model from a set of statements, across three problem types and two difficulty grades, while controlling for the response-position and label biases that let models shortcut similar tasks.",
  "metric": {
   "baseline_note": "The harness also reports standard error and a weighted accuracy that adjusts for tuple ID, problem name and problem size, to control for known response biases.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "WorldSense",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2311.15930",
   "title": "WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models",
   "url": "https://arxiv.org/abs/2311.15930",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Youssef Benchekroun",
    "Megi Dervishi",
    "Mark Ibrahim",
    "Jean-Baptiste Gaya",
    "Xavier Martinet",
    "Gr\u00e9goire Mialon",
    "Thomas Scialom",
    "Emmanuel Dupoux",
    "Dieuwke Hupkes",
    "Pascal Vincent"
   ],
   "org": "",
   "url": "https://arxiv.org/abs/2311.15930"
  },
  "released": "2023-11",
  "repo_url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/worldsense",
  "saturation": {
   "as_of": "",
   "note": "",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "worldsense task source (Inspect Evals)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/worldsense/worldsense.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "worldsense task README (Inspect Evals)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/worldsense/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "WorldSense: A Synthetic Benchmark for Grounded Reasoning in Large Language Models",
    "url": "https://arxiv.org/abs/2311.15930"
   }
  ],
  "status": "active",
  "subcategory": "reasoning",
  "summary": "WorldSense is a synthetic benchmark that tests whether a model can maintain a consistent world model while controlling for dataset bias.",
  "tags": [
   "benchmark",
   "reasoning",
   "synthetic",
   "bias-controlled"
  ],
  "task_format": "Text input describing a small scenario (e.g. object placements or a scheduling puzzle) with three problem types: Infer (judge a statement true or false), Compl (pick the correct statement among three options), and Consist (judge whether a set of statements is possible or impossible)."
 }
}