{
 "body": "\n## What it measures\n\nWeb of Lies presents a short word problem in which a chain of people each assert whether another\nperson tells the truth or lies, mathematically equivalent to composing a chain of negation and\nidentity functions, and asks whether the last person in the chain is telling the truth. Answering\nrequires tracking a boolean state correctly through several nested statements, testing multi-step\nlogical reasoning framed as a naturalistic narrative rather than an explicit formula.\n\n## How it is scored\n\nThe task is a balanced binary yes/no multiple-choice classification, scored with BIG-bench's\nmultiple_choice_grade metric (accuracy); chance performance is 0.5. Examples are generated\nprocedurally by the task's own code rather than drawn from a static file, producing chains of\nvarying length. No human baseline figure was found in the source read for this page.\n\n## Dataset and licence\n\nThe task holds 1,600 multiple-choice examples, generated procedurally at runtime by randomly\nassigning names and truth-or-lie statements to build chains of nested boolean functions. Because\ngeneration is procedural rather than from a fixed, hand-curated item file, there is no separate\nstatic dataset file with its own licence beyond the BIG-bench repository's own terms; no explicit\nlicence is stated for the task's generation code in the directory itself.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Roman Novak (Google) as part of the broader, multi-author\nBIG-bench collaboration. No separate standalone paper describing this task in isolation was found;\nit is documented in the task's own README and code, and separately appears in the BIG-bench Hard\npaper as one of the tasks selected for that harder subset.\n\n## Lineage\n\nWeb of Lies is one of several hundred independent BIG-bench tasks. It was later selected as one of\nthe 23 tasks making up BIG-bench Hard (BBH), a subset chosen because prior language models had not\nexceeded average human rater performance on it in the original BIG-bench evaluation; BBH does not\nhave its own page in this repository yet. The task has no predecessor or successor tracked here.\n\n## Saturation and contamination\n\nNo maintained standalone leaderboard for this task was found. Its inclusion in BIG-bench Hard\nsignals that, as of BIG-bench's original evaluation, no model tested had beaten average human\nperformance on it, but whether current frontier models have since closed that gap is not\nestablished from a source read for this page, so saturation status is marked \"watch.\" The task's\nown README documents a known shortcut, that the task can be solved by checking the parity of how\nmany times the word \"lies\" appears in the prompt without doing the underlying reasoning, which\nmeans a high score does not by itself confirm a model is performing genuine multi-step tracking.\nContamination risk is low relative to fixed-item BIG-bench tasks because examples are generated\nprocedurally at runtime rather than sampled from a static public file, though the task format and\nits documented shortcut have been publicly discussed since 2021.\n\n## How to run it\n\nRun as the `web_of_lies` task in the BIG-bench repository\n(`bigbench/benchmark_tasks/web_of_lies`), or via BIG-bench Hard. lm-evaluation-harness implements\nseveral BBH variants of it under its `bbh` task group (zero-shot, few-shot and chain-of-thought\nforms); the confirmed zero-shot task name is `bbh_zeroshot_web_of_lies`, which prompts with a plain\n\"Q: ... A:\" template and extracts a yes/no answer via regex matching phrases like \"tells the truth\"\nor \"does not tell the truth.\" Scores from the two source repositories, and across BBH's own\nvariant task names, are not directly comparable without checking shot count and whether\nchain-of-thought was used.\n\n## Reading the numbers\n\nA high score suggests a model can track a boolean state correctly through a chain of nested\ntruth-teller/liar statements, a simple form of multi-step logical reasoning. Because the task can\nin principle be solved by a documented shortcut (counting occurrences of \"lies\" and checking\nparity) rather than by explicitly tracking the chain, a high score alone does not confirm the model\nis doing the intended step-by-step reasoning; comparisons that also check performance on longer\nchains, where the shortcut still applies but explicit tracking gets harder, are more informative\nthan the raw accuracy number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "web_of_lies",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "Because examples are generated procedurally at runtime rather than drawn from a fixed, pre-written item bank, the exact chains of names and statements a model sees are unlikely to match items memorised from a fixed public file, lowering contamination risk relative to BIG-bench tasks built from static item sets. The general task format and its README (including the documented \"lies\" parity shortcut) have been public since 2021, so a model could still learn to exploit that shortcut from public discussion of the task rather than from seeing exact items.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1600,
   "size_note": "1,600 multiple-choice queries, per the task's own README and results for the reference (\"dummy\") model. Examples are generated procedurally by the task's own code, which builds random chains of truth-teller/liar statements of varying length rather than sampling from a static, hand-written question bank.\n",
   "splits": "Single BIG-bench task file of 1,600 procedurally generated examples; no separate train/validation/test split is defined by the task itself",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/web_of_lies"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-008 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-008"
  },
  "harness": {
   "bigbench": "web_of_lies",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "bbh_zeroshot_web_of_lies",
   "opencompass": "",
   "other": "Also ships as one of the 23 tasks in BIG-bench Hard (BBH). lm-evaluation-harness carries several BBH variants (zero-shot, few-shot, and chain-of-thought) under its `bbh` task group; the confirmed zero-shot task name is `bbh_zeroshot_web_of_lies`, which extracts a yes/no answer via regex matching phrases like \"tells the truth\" or \"does not tell the truth.\"\n"
  },
  "id": "web_of_lies",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Web of Lies gives a model a short narrative in which several people each state whether another person tells the truth or lies, forming a chain equivalent to a composition of negation and identity functions, f_n(f_{n-1}(...f_1(x)...)). The model must determine whether the final person in the chain is telling the truth, answering yes or no. It measures multi-step boolean reasoning and the ability to track state (truth-value) through a chain of statements, framed as a naturalistic word problem rather than an explicit logic formula.\n",
  "metric": {
   "baseline_note": "The task is a balanced binary yes/no choice, so chance performance is 0.5. No human baseline figure was found in the source read for this page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Multiple choice grade (accuracy on the yes/no final-truth-value question)",
   "random_baseline": 0.5,
   "unit": "accuracy"
  },
  "name": "Web of Lies",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Roman Novak"
   ],
   "org": "Google (BIG-bench collaboration); task author Roman Novak",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/web_of_lies"
  },
  "released": "2021",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/web_of_lies",
  "saturation": {
   "as_of": "",
   "note": "No maintained standalone leaderboard for this task was found for this page. It is included as one of the 23 tasks in BIG-bench Hard (BBH), a subset selected because the original BIG-bench paper found no prior language model beat average human rater performance on it, implying it was not saturated at BIG-bench's original 2022 evaluation. The task's own README also notes it can in principle be solved by a shortcut, checking the parity of how many times the word \"lies\" appears in the prompt, without doing the underlying reasoning, so a high score does not by itself confirm the intended reasoning is happening. Whether current frontier models have saturated it, and whether they rely on that shortcut, is not established from a source read for this page.\n",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench web_of_lies task directory (README: task description, boolean function composition framing, 1,600 examples, multiple_choice_grade metric, author, keywords, documented parity shortcut)",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/web_of_lies"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench web_of_lies task.py: procedural generation of random truth-teller/liar chains, binary yes/no scoring via p_yes vs p_no comparison",
    "url": "https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/web_of_lies/task.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness BBH zero-shot web_of_lies.yaml (task name bbh_zeroshot_web_of_lies, prompt template, yes/no regex answer extraction)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/bbh/zeroshot/web_of_lies.yaml"
   }
  ],
  "status": "active",
  "subcategory": "multi-step boolean logic (nested truth-teller/liar word problems)",
  "summary": "A BIG-bench task that phrases a chain of nested boolean functions as a word problem about people who tell the truth or lie, and asks the model to answer yes or no.",
  "tags": [
   "logical-reasoning",
   "multi-step-reasoning",
   "big-bench",
   "bbh"
  ],
  "task_format": "Zero-shot binary (yes/no) multiple-choice classification, scored with BIG-bench's multiple_choice_grade metric. Examples are generated procedurally at runtime by randomly assigning names and truth/lie statements to build a chain of a given length, rather than drawn from a fixed, pre-written item bank.\n"
 }
}