{
 "body": "\n## What it measures\n\nSymbol Interpretation (SIT) asks a model to reason about \"structures\": sequences of six pieces,\neach rendered as an emoji, standing in for objects in a small constructed world with positions and\nproperties. Given two such structures and a set of candidate sentences, the model must pick the\nsentence that is true of both. Correctly solving this requires binding emoji symbols to the roles\nor names they represent, then evaluating logical statements (quantifiers, boolean operators, and\npositional relations such as \"to the left of\") against that binding \u2014 a controlled test of\nsymbolic and visual-symbolic reasoning rather than general world knowledge.\n\nThe task is deliberately built as five parallel subtasks that probe robustness to how symbols are\ngrounded: a \"plain\" version with a direct, consistent emoji-to-meaning mapping; an \"adversarial\"\nversion that mismatches emojis and names; a \"tricky\" version with reversed descriptions; and two\n\"agnostic\" versions that strip away either the names or the emojis in favor of generic placeholders.\nComparing performance across these variants is meant to reveal whether a model is genuinely tracking\nthe logical structure or relying on surface priors about what particular emojis \"mean.\"\n\n## How it is scored\n\nEach subtask is scored with BIG-bench's `multiple_choice_grade` metric, and each is also its own\n`preferred_score`. Items in the \"plain\" subtask carry five answer options each (a `target_scores`\nobject with five keys, one marked correct), giving a uniform-random baseline of 0.2 for that\nsubtask. The task is zero-shot: items are drawn only from the single example set in each\nsubtask's own `task.json`, with no separate few-shot demonstration set described in the task files.\n\n## Dataset and licence\n\nThe task's top-level `task.json` is a manifest (description, keywords, canary) rather than the item\ndata; the actual items live in five subtask folders (`plain`, `adversarial`, `tricky`,\n`name_agnostic`, `emoji_agnostic`), each with its own `task.json` and `examples` array. The task\nREADME states a total of 990 multiple-choice queries (198 per subtask, over three difficulty\ntiers of 66); a direct count of the `plain` subtask's own examples array returned 198, confirming\nthat figure. All items and their correct answers are public in the repository, under the BIG-bench\nrepository's Apache-2.0 licence.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Antonio Norelli, Andrea Santilli, Giorgio Mariani, Luca\nMoschella, Giambattista Parascandolo, Simone Melzi and Emanuele Rodol\u00e0. No standalone paper\ndescribing this specific task was located; it is documented only within BIG-bench itself, described\nin \"Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models\"\n(arXiv:2206.04615, 2022; later published in Transactions on Machine Learning Research). The\n`google/BIG-bench` repository was archived by its owner on 2026-04-17 and is now read-only.\n\n## Lineage\n\nSymbol Interpretation is a standalone, self-contained BIG-bench task with five internal subtask\nvariants (plain, adversarial, tricky, name-agnostic, emoji-agnostic) rather than external\npredecessor or successor benchmarks. Its own keywords tag it as part of \"BIG-bench Lite,\" the\ncurated, smaller subset of BIG-bench tasks used for cheaper evaluation runs, though no separate\nBIG-bench Lite leaderboard entry specific to this task was independently confirmed here.\n\n## Saturation and contamination\n\nNo dedicated leaderboard or paper table reporting per-model scores on this task, or its individual\nsubtasks, was located, so saturation status is unknown; the \"out of distribution\" keyword in its own\nmetadata suggests the task's authors intended the adversarial and agnostic variants to remain\ndifficult even as models improve on the plain variant. Contamination risk is judged medium: all\nitems and answers are public and BIG-bench has been indexed since 2022, though the emoji-based,\nsynthetic-world framing is less likely to appear verbatim in general web text than more naturalistic\nQA content.\n\n## How to run it\n\nThe canonical implementation is the task directory in the archived `google/BIG-bench` repository\n(`bigbench/benchmark_tasks/symbol_interpretation`), with each of the five subtasks runnable\nseparately through BIG-bench's own task-running code. It was not found among the tasks reimplemented\nin EleutherAI's lm-evaluation-harness `bigbench` multiple-choice task set, so reproducing scores\nrequires the original `bigbench` Python package or a direct reimplementation from each subtask's\n`task.json`.\n\n## Reading the numbers\n\nA high score on the \"plain\" subtask shows a model can track simple logical and positional relations\nonce symbol-to-meaning mapping is unambiguous; a much lower score on \"adversarial,\" \"tricky,\" or the\n\"agnostic\" variants under the same model would suggest the plain-subtask score partly reflects\nsurface pattern-matching on familiar emoji-name pairings rather than robust symbolic reasoning.\nBecause no cross-model leaderboard for this task could be confirmed here, compare subtask scores\nagainst each other for the same model rather than treating any single number as an externally\ncalibrated capability measure.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "symbol_interpretation",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SIT"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "All five subtasks' items and correct answers are public in the BIG-bench GitHub repository, which has been indexed since 2022, so the item text is plausibly present in web-scale pretraining data. The task carries a BIG-bench canary string requesting exclusion from training corpora, which is voluntary and unverifiable from this page.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 990,
   "size_note": "The task README states the benchmark totals 990 multiple-choice queries across five subtasks (plain, adversarial, tricky, name-agnostic, emoji-agnostic), describing each subtask as 198 examples split across three difficulty tiers of 66. A direct count of the \"plain\" subtask's own task.json examples array returned 198 items, confirming the README's per-subtask figure and the 990 total (198 x 5).\n",
   "splits": "five subtask folders (plain, adversarial, tricky, name_agnostic, emoji_agnostic), each a single set with no train/test split",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/symbol_interpretation"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-007 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-007"
  },
  "harness": {
   "bigbench": "symbol_interpretation",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "symbol_interpretation",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The model is given a \"structure\": a sequence of six pieces represented by emojis, standing in for objects in a simple constructed world. It must choose, from a set of candidate sentences, the one that correctly and consistently describes two given structures. The task is split into five subtasks that vary how directly the emojis map to their described meaning: a \"plain\" version with direct emoji-to-name correspondence, an \"adversarial\" version with intentionally mismatched emoji-name associations, a \"tricky\" version with reversed object descriptions, and two \"agnostic\" versions that substitute generic placeholders for either the names or the emojis. Within each subtask, items escalate across difficulty tiers covering simple quantification, logical operators, and positional relationships between pieces.\n",
  "metric": {
   "baseline_note": "Each subtask's task.json lists \"multiple_choice_grade\" as its metric and preferred_score. A direct read of the \"plain\" subtask's examples confirms five answer options per item (target_scores with five keys, one scored 1), giving a uniform-random baseline of 0.2.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "multiple_choice_grade",
   "random_baseline": 0.2,
   "unit": ""
  },
  "name": "Symbol Interpretation",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Antonio Norelli",
    "Andrea Santilli",
    "Giorgio Mariani",
    "Luca Moschella",
    "Giambattista Parascandolo",
    "Simone Melzi",
    "Emanuele Rodol\u00e0"
   ],
   "org": "Google (BIG-bench collaboration)",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/symbol_interpretation"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/symbol_interpretation",
  "saturation": {
   "as_of": "",
   "note": "No dedicated leaderboard or paper table with per-model scores for this task was located; it is tagged 'BIG-bench Lite' in its own keywords, meaning it belongs to BIG-bench's curated lightweight subset, but no specific score table for it was independently confirmed.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench: symbol_interpretation task directory",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/symbol_interpretation"
   },
   {
    "accessed": "2026-09-08",
    "title": "symbol_interpretation README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/symbol_interpretation/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "symbol_interpretation root task.json (manifest)",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/symbol_interpretation/task.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "symbol_interpretation/plain subtask task.json",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/symbol_interpretation/plain/task.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository (archived 2026-04-17)",
    "url": "https://github.com/google/BIG-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository LICENSE",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
    "url": "https://arxiv.org/abs/2206.04615"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness bigbench multiple_choice task directory",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/bigbench/multiple_choice"
   }
  ],
  "status": "unknown",
  "subcategory": "structured visual/logical reasoning via emoji-symbol interpretation (BIG-bench Lite)",
  "summary": "BIG-bench Lite task: pick which sentence correctly describes a 'structure' \u2014 a sequence of six emoji pieces \u2014 across five adversarial variants.",
  "tags": [
   "bigbench-lite",
   "visual-reasoning",
   "logical-reasoning",
   "emoji",
   "multiple-choice",
   "out-of-distribution"
  ],
  "task_format": "Multiple-choice, zero-shot; each item asks which sentence is consistent with two given emoji structures."
 }
}