{
 "body": "\n## What it measures\n\nTaboo is a BIG-bench \"programmatic\" task that turns the party game Taboo into a two-phase\nmodel-vs-model evaluation. For each of 100 target words, the task supplies a list of closely\nrelated \"taboo\" words (selected via word embeddings) that a definer must avoid. One model instance\nplays the definer: it must describe the target word well enough for a listener to guess it, without\nusing any of the forbidden taboo words. A second model instance (which never sees the target word)\nplays the guesser: it reads only the definition and must name the target concept.\n\nThis jointly exercises constrained creative generation \u2014 finding a way to communicate a concept\nwhile avoiding the words that would make it easy \u2014 and inference under indirection, since the\nguesser must recover a specific target from an intentionally roundabout description. Difficulty can\nbe tuned by changing how many taboo words are forbidden.\n\n## How it is scored\n\n`task.py` scores the definer's response by subtracting one point for every forbidden word it uses\n(so violating the constraint directly lowers the score), and scores the guesser's response as\n1 divided by the length of its response if the target word appears in it, or 0 if it does not,\nrewarding short, confident, correct guesses over long ones. These two phase-scores are combined\ninto an overall score. The default configuration constrains the definer to avoid `k=5` taboo words\nout of a configurable maximum of 10, giving an achievable range from roughly `-k` up to 1.0. No\nrandom-guess or human-performance baseline is published for this task.\n\n## Dataset and licence\n\n`taboo_data.json` contains 100 examples, each an object with a `target` word and its list of\n`taboos`. The task README separately describes the benchmark as containing \"200 free text queries,\"\nconsistent with each of the 100 items producing two prompts at evaluation time \u2014 one for the\ndefiner role and one for the guesser role. All target words and their taboo lists are visible in\nthe public JSON file, under the BIG-bench repository's Apache-2.0 licence.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Dar Gilboa. BIG-bench itself is described in \"Beyond the\nImitation Game: Quantifying and extrapolating the capabilities of language models\"\n(arXiv:2206.04615, 2022; later published in Transactions on Machine Learning Research), a\nlarge multi-author collaboration coordinated by Google researchers. The `google/BIG-bench`\nrepository was archived by its owner on 2026-04-17 and is now read-only, so no further maintenance\nor leaderboard updates should be expected from that source.\n\n## Lineage\n\nTaboo is a standalone BIG-bench task with no stated predecessor or successor. It belongs to a small\nfamily of BIG-bench \"self-play\" or \"repeated interaction\" tasks (per its own keywords) in which a\nmodel or pair of model instances interacts with itself rather than answering a single static\nquestion, distinguishing it from BIG-bench's more numerous static multiple-choice tasks; no other\ntask in this repository is currently identified as a direct variant of it.\n\n## Saturation and contamination\n\nNo dedicated leaderboard or paper table reporting per-model scores on this task was located, so\nsaturation status is unknown. Because the task is a tunable, programmatic self-play evaluation\n(scores depend on the chosen taboo-word-count parameter `k` and on which two model calls play\ndefiner versus guesser), scores are also not automatically comparable across different evaluation\nconfigurations. Contamination risk is judged low: even if a model has seen `taboo_data.json`\nverbatim, doing well still requires generating a fresh, valid constrained definition and making a\ncorrect live inference from it, rather than reproducing a memorized answer.\n\n## How to run it\n\nThe canonical implementation is the task directory in the archived `google/BIG-bench` repository\n(`bigbench/benchmark_tasks/taboo`), which uses a custom `task.py` (a \"programmatic\" BIG-bench task\nrather than a plain JSON multiple-choice task) to orchestrate the two-phase definer/guesser\ninteraction and compute the combined score. It was not found among the tasks reimplemented in\nEleutherAI's lm-evaluation-harness `bigbench` task set, so reproducing it requires the original\n`bigbench` Python package, including its self-play orchestration logic.\n\n## Reading the numbers\n\nA high combined score indicates a model can both generate a useful, constraint-respecting\ndefinition and correctly infer a target concept from another model's indirect description \u2014 a\nproxy for constrained generation and pragmatic language understanding rather than for stored\nfactual knowledge. Because the game is self-play, a strong score can also depend on whether the\nsame model plays both roles or a different model plays the guesser, and on the specific\ntaboo-word-count setting used, so any reported figure should be read together with those evaluation\ndetails rather than as a single universal number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "taboo",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "generation",
  "contamination": {
   "note": "The task score depends on live, generated multi-turn interaction between two model instances (or roles) rather than on recalling a fixed gold text, so memorizing taboo_data.json's target and taboo-word lists would not by itself guarantee a good score; the model must still generate a valid constrained definition and correctly infer the target from it at evaluation time.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 100,
   "size_note": "taboo_data.json contains 100 examples, each pairing one target word with a list of associated taboo (forbidden) words drawn from word embeddings. The task README additionally describes the benchmark as \"0 multiple choice and 200 free text queries,\" consistent with each of the 100 target-word items generating two free-text queries (one prompting the definer, one prompting the guesser).\n",
   "splits": "single set of 100 target-word items, no train/test split",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/taboo"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-007 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-007"
  },
  "harness": {
   "bigbench": "taboo",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "taboo",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Taboo implements the party game of the same name as a two-model self-play task. Given a target word and a list of \"taboo\" words closely related to it (drawn from word embeddings), one model instance must produce a definition of the target that avoids using any of the taboo words; a second model instance then reads only that definition and must guess the target word. The task jointly tests constrained, creative language generation (defining a concept under vocabulary restrictions) and language comprehension (inferring a specific concept from an indirect description), and can be run at different difficulty levels by varying how many taboo words are forbidden.\n",
  "metric": {
   "baseline_note": "task.py scores the first (defining) response by subtracting one point for each forbidden (\"taboo\") word it uses, and scores the second (guessing) response as 1 divided by the guesser's response length if the target word appears in it, 0 otherwise. The two phase-scores are combined (the task documentation describes summing/averaging them). With a default constraint level of k=5 forbidden words (out of a configurable max_k=10), the achievable score range runs from -k up to 1.0; no random-guess or human baseline is published.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "combined score (definition penalty + guesser reward)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Taboo",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Dar Gilboa"
   ],
   "org": "Google (BIG-bench collaboration)",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/taboo"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/taboo",
  "saturation": {
   "as_of": "",
   "note": "No dedicated leaderboard or paper table reporting per-model scores on this task was located; as a programmatic self-play task with a tunable difficulty constant (k), its scores also are not necessarily comparable across evaluations run at different k values.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench: taboo task directory",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/taboo"
   },
   {
    "accessed": "2026-09-08",
    "title": "taboo README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/taboo/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "taboo task.py implementation",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/taboo/task.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "taboo_data.json",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/taboo/taboo_data.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository (archived 2026-04-17)",
    "url": "https://github.com/google/BIG-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository LICENSE",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "unknown",
  "subcategory": "constrained definition generation and comprehension, two-agent self-play word game",
  "summary": "BIG-bench self-play game: one model instance must define a target word without using forbidden related words, and a second instance must guess it.",
  "tags": [
   "bigbench",
   "self-play",
   "constrained-generation",
   "word-game",
   "programmatic-task"
  ],
  "task_format": "Free-response, two-phase, programmatic self-play: model A generates a definition under a word-avoidance constraint, model B (or the same model) guesses the target from that definition."
 }
}