{
 "body": "\n## What it measures\n\nTwenty Questions makes two copies of a model communicate a hidden concept. Alice sees the concept and answers Bob's questions with only yes or no. Bob chooses questions and eventually names the concept.\n\nThe task combines question generation, constrained answering, consistency, self-play coordination and persistence across a long interaction. It measures an interactive protocol rather than static question answering.\n\n## How it is scored\n\nThe README defines the score as the negative conversational round in which Bob guesses correctly. A guess at round five receives -5. If Bob has not guessed by question 100, the score is -100. Less negative (higher) values indicate earlier success, and the reference task.py fixes the aggregate score's bounds at low_score=-100 and high_score=-10. No random or human baseline is established.\n\nBob is not told the concept, while Alice is. Alice may answer only yes or no, and Bob should ask one question per turn. Allowing explanations, multiple questions or another maximum length changes the task.\n\n## Dataset and licence\n\nThis is a procedurally described game rather than a fixed question table. The generated header reports 3,708 multiple-choice and 3,713 free-text dummy-model queries, but the README does not establish these as a stable item split. A separate data licence is not stated.\n\n## Who publishes it\n\nGoogle's BIG-bench repository hosts the task and credits Jascha Sohl-Dickstein. The 2022 BIG-bench paper provides broader context. No current standalone leaderboard was established.\n\n## Lineage\n\nThe task is related to conversational QA work such as QuAC and CoQA, which the README cites, but it is not a subset of either. No successor or repository variant was established.\n\n## Saturation and contamination\n\nThe README reports failures from models up to 128 million parameters in early experiments. That does not establish a current frontier ceiling. The task and canary are public, and no private or rotating concept policy is described.\n\n## How to run it\n\nRun the BIG-bench task twenty_questions with two model instances. Preserve the role prompts, one-line turn format, yes-or-no restriction and 100-question cap. Record transcripts because protocol violations affect the result.\n\n## Reading the numbers\n\nA score near zero means Bob identified the concept quickly. A score near -100 means the interaction failed within the cap. The number mixes question quality, Alice's consistency and protocol compliance. Report settings and inspect transcripts alongside the score.\n\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "twenty_questions",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "BIG-bench Twenty Questions"
  ],
  "category": "agentic",
  "contamination": {
   "note": "The task logic and canary are public; no private concept policy is described.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The task is procedurally described; a stable item count is not stated.",
   "splits": "interactive task; no fixed train/test split stated",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/twenty_questions"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-new-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-003"
  },
  "harness": {
   "bigbench": "twenty_questions",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "twenty_questions",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "big_bench",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Alice receives a hidden concept and answers yes or no. Bob asks questions and must identify it. The task tests constrained answering, targeted question selection, persistence and self-play.",
  "metric": {
   "baseline_note": "task.py sets low_score=-100 and high_score=-10 for the averaged per-concept score; less negative (higher) values are better. No random or human baseline is given.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": -10,
   "name": "negative number of conversational rounds to the correct guess",
   "random_baseline": null,
   "unit": "rounds"
  },
  "name": "Twenty Questions",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Jascha Sohl-Dickstein"
   ],
   "org": "BIG-bench collaboration",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/twenty_questions"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "Historical small-model failures do not establish a current ceiling.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench Twenty Questions README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/twenty_questions/README.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench Twenty Questions task.py",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/twenty_questions/task.py"
   }
  ],
  "status": "active",
  "subcategory": "self-play concept identification",
  "summary": "Two model instances play Twenty Questions, communicating a hidden concept through yes-or-no answers.",
  "tags": [
   "big-bench",
   "agentic",
   "self-play",
   "interactive"
  ],
  "task_format": "Interactive two-agent text game with one question per turn and a maximum of 100 questions."
 }
}