{
 "body": "\nPart of the [BIG-bench](big_bench.md) family.\n\n## What it measures\n\nmnist_ascii renders MNIST digit images as ASCII art and asks the model to identify the digit. The pattern is text on a page, but success requires reading its spatial shape. It is a fixed BIG-bench task rather than a conventional image-input benchmark.\n\n## Reading the numbers\n\nThere are ten digit choices, so chance is 10%. A high score means the model recognized the rendered digit under this font and formatting. It does not establish general image recognition or robustness to another ASCII renderer. The task README reports that Google-internal models failed, but no current frontier score was established here.\n\nFormatting is part of the task definition: changing the font, spacing, or line width can change the visual signal. Scores should therefore be compared only when the renderer and prompt wrapper are held constant. The benchmark also says little about recognizing natural images because the input contains no pixels.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mnist_ascii",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "BIG-bench mnist_ascii"
  ],
  "category": "multimodal",
  "contamination": {
   "note": "The task is public and contains fixed ASCII renderings. BIG-bench embeds a canary string, but no measured contamination study was opened.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 69984,
   "size_note": "The BIG-bench task README header reports 69,984 multiple-choice queries.",
   "splits": "Single fixed task set; no train/test split",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/mnist_ascii"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-batch-059 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-batch-059"
  },
  "harness": {
   "bigbench": "mnist_ascii",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "mnist_ascii",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "big_bench",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The task converts MNIST digit images into ASCII art and asks which digit is shown. The input is text arranged as a visual pattern, so the task probes whether a language model can switch from language processing to simple visual layout reading.\n",
  "metric": {
   "baseline_note": "Ten digit labels imply 10% uniform-choice chance; no human score was published in the task README.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "multiple_choice_grade",
   "random_baseline": 10.0,
   "unit": "%"
  },
  "name": "ASCII MNIST",
  "page_kind": "subset",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Ethan Dyer",
    "Adam R. Brown"
   ],
   "org": "Google BIG-bench",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/mnist_ascii"
  },
  "released": "2021",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/mnist_ascii",
  "saturation": {
   "as_of": "",
   "note": "The task README says Google-internal models failed, but no current leaderboard cell was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench mnist_ascii README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/mnist_ascii/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench Apache License",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "unknown",
  "subcategory": "ASCII-art digit recognition",
  "summary": "A BIG-bench task that asks a model to identify MNIST digits rendered as ASCII art.",
  "tags": [
   "big-bench",
   "subset",
   "ascii",
   "mnist",
   "visual-reasoning"
  ],
  "task_format": "Ten-way multiple choice over digit labels 0 through 9; BIG-bench preferred metric is multiple_choice_grade."
 }
}