{
 "body": "\n## What it measures\n\nText Navigation Game asks a model to interpret an ASCII-grid representation of a maze and navigate to a target using free-text move commands. Unlike a single-shot question, the model plays out a game: each turn it sees the current grid and issues a command, the environment updates the grid, and this repeats for up to 30 turns per game.\n\nThe task is procedurally generated (`task.py`, default seed 42) over four board sizes (5, 9, 12, 15) with 20 games per size by default. BIG-bench tags it \"visual reasoning,\" \"repeated interaction\" and \"common sense,\" reflecting that the model must track a spatial structure over multiple turns, not just parse one instruction.\n\n## How it is scored\n\nThe task's own code (`TextNavigationTask.evaluate_model`) reports three metrics per board size: `targets_reached` (fraction of games where the model reached the target within 30 turns, the preferred score), `valid_queries` (fraction of moves that parsed as a valid movement or help request), and `good_queries` (fraction of moves that made progress). It is not multiple-choice grading. No human baseline is stated in the code.\n\n## Dataset and licence\n\nThe task has no fixed `task.json` item list; it is generated at run time by `task.py` from a seed and configurable map sizes and games-per-size. With the default configuration this yields 80 games. The BIG-bench repository does not state a separate licence for the task. The generation code and default seed are public, so exact game layouts are reproducible and exposed.\n\n## Who publishes it\n\nText Navigation Game is a Google BIG-bench task authored by C. Daniel Freeman and covered by the broader BIG-bench paper. No separate paper or current leaderboard was established.\n\n## Lineage\n\nThis is a standalone BIG-bench task. No predecessor, successor, or formal variant was established.\n\n## Saturation and contamination\n\nSaturation is unknown; no leaderboard reports scores. The README notes that early spot-testing found GPT-3 and Google-internal models performed \"significantly worse\" than an agent moving randomly. Because the generator and default seed are public, the exact game layouts can be memorized; contamination risk is medium.\n\n## How to run it\n\nRun BIG-bench task `text_navigation_game`. Preserve the repository revision, the seed, the map sizes and games-per-size configuration, and the 30-turn limit, since all of these change the effective difficulty and the resulting `targets_reached` score. The task directory (`task.py`) is the primary runnable source; there is no static task file to diff against.\n\n## Reading the numbers\n\nA high `targets_reached` score indicates the model can parse an ASCII maze, issue valid move commands, and navigate to a target over several turns. It does not establish spatial reasoning with images, robotics, or unfamiliar environments, and it says nothing about single-turn question answering. Inspect `valid_queries` and `good_queries` alongside `targets_reached`: a model can fail to reach the target while still issuing valid, sensible moves, or it can produce many unparseable moves and never reach the target at all.\n\nBecause the task is a multi-turn game rather than a fixed item set, report the exact map sizes, games-per-size, seed, and turn limit used, since these directly change task difficulty and are not fixed by the benchmark itself.\n\nScores should be treated as narrow evidence about interactive, text-based spatial navigation. They should not be generalized to physical navigation, planning, or visual grid tasks presented as images.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "text_navigation_game",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "The task is public; exposure is unknown.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "The task is procedurally generated by task.py (default seed 42, map sizes 5/9/12/15, 20 games per size = 80 games per full run), not a fixed task.json item list.",
   "splits": "",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/text_navigation_game"
  },
  "freshness": {
   "luna-new-001 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-001"
  },
  "harness": {
   "bigbench": "text_navigation_game",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "text_navigation_game",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "bigbench",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The task evaluates whether a model can interpret a text/ASCII representation of a grid world, issue valid movement commands over repeated turns, and reach a target location. It is a small procedurally generated, multi-turn reasoning task, not a single-shot question.",
  "metric": {
   "baseline_note": "The task also reports valid_queries and good_queries (fractions of parseable and progress-making moves); no human baseline is stated in the task code.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "targets_reached",
   "random_baseline": null,
   "unit": "fraction"
  },
  "name": "Text Navigation Game",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "C. Daniel Freeman"
   ],
   "org": "Google BIG-bench",
   "url": "https://github.com/google/BIG-bench"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "No current leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench task directory",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/text_navigation_game"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench Text Navigation Game README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/text_navigation_game/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench Text Navigation Game task implementation",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/text_navigation_game/task.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "active",
  "subcategory": "navigation",
  "summary": "BIG-bench Text Navigation Game has a model issue free-text moves across turns to reach a target on an ASCII-grid maze.",
  "tags": [
   "navigation",
   "reasoning",
   "synthetic",
   "multi-turn",
   "programmatic"
  ],
  "task_format": "Repeated interaction over up to 30 turns. The model receives the current ASCII-grid game state and issues a free-text move command each turn; the environment updates state and reports whether the target was reached."
 }
}