{
 "body": "\n## What it measures\n\nBIG-bench defines the task as determining the final positions of a set of objects, given their initial positions and a natural-language description of a sequence of pairwise swaps (for example, people exchanging balls, gifts, or dance partners). Its keywords are logical reasoning, visual reasoning, multi-step, decomposition, and multiple choice, confirmed directly from the task metadata.\n\n## How it is scored\n\nThe task is multiple choice over the possible final holders of an object; the base BIG-bench metadata does not establish a separate normalization or baseline. BIG-Bench Hard (BBH), a widely used harder subset, includes three difficulty variants of this same task -- `tracking_shuffled_objects_three_objects`, `_five_objects`, and `_seven_objects` -- each with 250 items (confirmed via the Hugging Face datasets-server), scored the same way but with more objects and swaps to track as the variant name increases.\n\n## Dataset and licence\n\nThe official task metadata gives the description and keywords but not a total item count or licence for this task release. It carries a canary warning against training-data inclusion. The BBH repository's three difficulty variants add 250 items each (750 items total across the three variants) and carry the same canary GUID.\n\n## Who publishes it\n\nGoogle BIG-bench maintains the task repository. No separate paper was published for this task specifically; the BBH subset that includes its three harder variants comes from Suzgun et al., \"Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them\" (2022, arXiv:2210.09261). No current standalone leaderboard was established for this task page.\n\n## Lineage\n\nThis is a BIG-bench task. [BIG-Bench Hard](bbh.md) selected its three harder difficulty variants (three, five, and seven objects) for inclusion in its 23-task suite because, at the time of BBH's construction, no evaluated model outperformed the average human rater on them. No other predecessor or successor was established.\n\n## Saturation and contamination\n\nSaturation is unknown. The task is public and its metadata includes a canary, so contamination risk cannot be quantified from the source.\n\n## How to run it\n\nUse the BIG-bench task name `tracking_shuffled_objects` with the BIG-bench runner. Record task revision and answer format.\n\n## Reading the numbers\n\nA strong score indicates reliable tracking of symbolic swaps in this task format. It does not establish general spatial reasoning or robustness to longer sequences. Compare task revision and prompting protocol.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "tracking_shuffled_objects",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "The task metadata includes a canary warning; exposure of its test items is a material concern.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "BIG-bench task metadata does not state one fixed aggregate item count.",
   "splits": "unknown",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/tracking_shuffled_objects"
  },
  "freshness": {
   "luna-new-002": null,
   "luna-new-002 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review"
  },
  "harness": {
   "bigbench": "tracking_shuffled_objects",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "tracking_shuffled_objects",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "bbh"
   ],
   "variants": []
  },
  "measures": "The task gives initial object positions and textual swaps, then asks for the final positions. It exercises symbolic tracking and multi-step reasoning.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "Tracking Shuffled Objects",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "Google BIG-bench",
   "url": "https://github.com/google/BIG-bench"
  },
  "released": "2021",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "No authoritative current leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench task metadata",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/tracking_shuffled_objects/task.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-Bench Hard task directory (three tracking_shuffled_objects difficulty variants)",
    "url": "https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main/bbh"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face datasets-server row count for the three_objects BBH variant",
    "url": "https://datasets-server.huggingface.co/size?dataset=lukaemon/bbh&config=tracking_shuffled_objects_three_objects"
   }
  ],
  "status": "active",
  "subcategory": "multi-step object tracking",
  "summary": "BIG-bench task for tracking object positions through a sequence of swaps.",
  "tags": [
   "reasoning",
   "multiple-choice"
  ],
  "task_format": "Multiple-choice answer to a swap sequence."
 }
}