{
 "body": "\n## What it measures\n\nZeroBench pairs one or more images with a question that demands multi-step visual reasoning: careful\ncounting, spatial relations, fine visual detail, or cross-referencing several parts of an image at once,\nrather than a single glance-and-answer. The 100 main questions were written by a pool of more than 20\nhuman question creators, then adversarially filtered -- any candidate a baseline model answered\ncorrectly was discarded -- so the surviving set was, by construction, unsolved by every model in the\nauthors' 20-model baseline at release. Most main questions (93 of 100) use a single image; most images\n(70 of 100) are ordinary photographs rather than synthetic compositions. Because a 100-question,\nall-or-nothing set gives little signal about partial progress, each main question also carries several\ncompanion \"subquestions\" (334 in total, averaging 3.3 per question) covering the intermediate reasoning\nsteps.\n\n## How it is scored\n\nBoth main questions and subquestions are graded by exact string match: the model must place its answer\nin curly braces, like \"{final_answer}\", and that text is compared directly against the reference answer,\nwith no partial credit. The authors chose exact match deliberately, having found no distance-based metric\nthat fairly covered the diversity of valid answer formats across 100 hand-written questions; they also\nexcluded binary, multiple-choice and small-integer (under 10) answers during curation specifically to\nkeep guessing from inflating scores. Because thinking-style models often perform best at higher, fixed\nsampling temperatures, the authors track three metrics rather than one: pass@1 (a single deterministic\nattempt), pass@5 (correct on at least one of five stochastic samples) and pass^5 (correct on all five) --\nand report only the latter two for models evaluated after the initial release, since single-sample pass@1\nbecomes unstable for high-temperature models.\n\n## Dataset and licence\n\n100 main questions survived a three-step pipeline that started from 140 candidates: difficulty feedback\nagainst early baseline models, then human review for answerability, formatting and sufficient difficulty\n(106 questions passed this step), then adversarial filtering that discarded any question a baseline\nmodel got right (the strongest scored just 4 of 106 before that final cut). 334 subquestions were\nwritten alongside this process. The GitHub repository is released under the MIT licence; the Hugging\nFace dataset repository is access-gated and carries a canary string to keep it out of future training\ncorpora, and this page did not find a separate published licence tag for the dataset content itself.\n\n## Who publishes it\n\nZeroBench was introduced by Jonathan Roberts and 33 co-authors, led from the University of Cambridge with\ncontributors at numerous other institutions, and posted to arXiv in February 2025; the paper was accepted\nat ICML 2026. The authors maintain the GitHub repository, the gated Hugging Face dataset, and a public\nproject page and leaderboard at zerobench.github.io, which they have continued updating -- including an\nevaluation-protocol correction in August 2026 -- well past the original release.\n\n## Lineage\n\nZeroBench names no direct predecessor or successor in this repository, but the paper positions itself\nagainst a line of \"difficulty-first\" benchmarks built to resist quick saturation -- ARC-AGI, GPQA, GRAB,\nHumanEval-V, VibeEval and the concurrently developed Humanity's Last Exam -- arguing those eroded faster\nthan intended and that ZeroBench's adversarial filtering was designed to hold up longer. No successor\nbenchmark building directly on ZeroBench was identified.\n\n## Saturation and contamination\n\nAt release, all 20 evaluated baselines scored 0% on both pass@1 and pass^5; the single best, Gemini 2\nFlash Thinking, managed 5% pass@5. Tracking 26 further models over the following year, the paper reports\nstate-of-the-art climbing to 19% pass@5 (Gemini 3 Pro) and 6% pass^5 (GPT-5.2), with 47% of the 100 main\nquestions solved at least once by at least one of the 46 models evaluated cumulatively -- real movement,\nbut still far from saturated on a 100-point scale. Contamination risk is medium: the dataset carries a\ncanary string and gated access specifically to deter training-set inclusion, but the authors themselves\nnote this \"is no guarantee,\" and 18 months of public discussion and red-teaming since release make some\nexposure plausible even so; the benchmark's comparatively slow score growth over that period is the\nauthors' own partial evidence against severe leakage.\n\n## How to run it\n\nNo major third-party harness was found to carry ZeroBench. The authors' GitHub repository is the\nreference implementation: a fixed zero-shot chain-of-thought prompt ending in an explicit\nbracketed-answer instruction, and exact-match grading code after parsing that bracket. Reported scores\ndiffer by decoding regime (one greedy pass for pass@1 versus five stochastic samples at temperature 0.7\nfor pass@5 and pass^5), and for models released after the original evaluation, only pass@5 and pass^5\nare typically reported, so a bare \"ZeroBench score\" without a stated metric name is ambiguous.\n\n## Reading the numbers\n\nA non-zero ZeroBench score is meaningful on its own, given the benchmark was built so no evaluated model\ncould score at release; a pass^5 score above zero specifically means a model answered a question\ncorrectly and consistently across five independent attempts, not just once by chance. Because pass@1\nbecomes unreliable for high-temperature \"thinking\" models, prefer pass@5 and pass^5 when comparing\nrecent models, and check which of the three metrics a reported number uses before comparing it to\nanother. Even the strongest scores remain well below the human baseline's already-low 29.5% mean, so\ntreat any ZeroBench number as a measure of remaining headroom, not near-human capability.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "zerobench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Muse Spark",
   "model_id": "meta/muse-spark",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 33.0,
   "source": "meta-blog, officechai, artificial-analysis"
  }
 ],
 "page": {
  "aliases": [],
  "category": "multimodal",
  "contamination": {
   "note": "The dataset ships a canary string (a superset of the BIG-bench canary string) specifically so benchmark data can be filtered out of future training corpora, and the Hugging Face repository is access-gated rather than freely downloadable. The authors are explicit that \"canary strings are intended to discourage direct training on our benchmark, but [are] no guarantee.\" The benchmark's empirically slow saturation over its first 18 months is offered by the authors as partial evidence against severe contamination, though the questions and images have still been publicly discussed and red-teamed by outside users since release.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": false,
   "size": 100,
   "size_note": "100 main questions, drawn from a pool of 140 candidate questions narrowed through a three-step pipeline: difficulty feedback against early baselines, human review for answerability and format (106 questions passed this step), then adversarial filtering that discarded any question a baseline model answered correctly (the strongest baseline scored just 4 of 106 before this final cut). A further 334 subquestions (about 3.3 per main question) were written during review to break each main question into its component reasoning steps. 93 of the 100 main questions use a single image, 7 use multiple images; 70 use natural (photographic) images and 30 use synthetic ones.\n",
   "splits": "main questions (100) + subquestions (334); a single evaluation set, no train/test division",
   "url": "https://huggingface.co/datasets/jonathan-roberts1/zerobench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice K",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "No major third-party harness was found to carry ZeroBench. The authors' GitHub repository publishes the reference prompt (a zero-shot chain-of-thought instruction ending in an explicit \"{final_answer}\" format requirement) and grading code (exact match after parsing the bracketed answer). Two decoding regimes are used: greedy, seeded decoding for a single deterministic pass@1 attempt, and stochastic decoding (temperature 0.7, top-p 0.95) sampled five times for pass@5 and pass^5. Because recent \"thinking\" models often run best at fixed, higher temperatures, the authors report only pass@5 and pass^5, not pass@1, for models evaluated after the initial release.\n"
  },
  "id": "zerobench",
  "last_updated": "2026-08",
  "leaderboard_url": "https://zerobench.github.io/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ZeroBench pairs one or more images with a question requiring multi-step visual reasoning -- careful counting, spatial relations, fine detail extraction, cross-referencing several parts of an image -- built by a pool of more than 20 human question creators and then adversarially filtered: any candidate question that a baseline model answered correctly was discarded. The surviving 100 \"main\" questions are ones no model in the authors' 20-model baseline set solved at release. Because a set of yes/no answers this hard would still be impossible to fully differentiate, each main question also has, on average, 3.3 companion \"subquestions\" (334 in total) covering the intermediate reasoning steps, which let a model that fails the main question still show partial progress. Images are mostly natural photographs (70 of 100) rather than synthetic compositions (30 of 100), and mostly single-image questions (93 of 100, with 7 requiring more than one image).\n",
  "metric": {
   "baseline_note": "29.5% (standard deviation 27.1) is the paper's human baseline, from 15 evaluators (undergraduate and postgraduate students) each working a partition of the 100 questions with only the image(s) and question text, no external resources; inter-rater reliability across five duplicated question sets was a Pearson correlation of 0.82. The very high standard deviation reflects genuinely uneven difficulty across question sets (some near 0%, some near-perfect) rather than inconsistent grading, per the authors. No fixed random-guess baseline is established, since answers are open-ended free text and the authors deliberately excluded guessable answer formats.\n",
   "direction": "higher_is_better",
   "human_baseline": 29.5,
   "max_score": 100,
   "name": "accuracy (pass@1 / pass@5 / pass^5)",
   "unit": "%"
  },
  "name": "ZeroBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2502.09696",
   "title": "ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models",
   "url": "https://arxiv.org/abs/2502.09696",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Jonathan Roberts",
    "Mohammad Reza Taesiri",
    "Ansh Sharma",
    "Akash Gupta",
    "Samuel Roberts",
    "Ioana Croitoru",
    "Simion-Vlad Bogolin",
    "Jialu Tang",
    "Florian Langer",
    "Vyas Raina",
    "Vatsal Raina",
    "Hanyi Xiong",
    "Vishaal Udandarao",
    "Jingyi Lu",
    "Shiyang Chen",
    "Sam Purkis",
    "Tianshuo Yan",
    "Wenye Lin",
    "Gyungin Shin",
    "Qiaochu Yang",
    "Anh Totti Nguyen",
    "David I. Atkinson",
    "Aaditya Baranwal",
    "Alexandru Coca",
    "Mikah Dang",
    "Sebastian Dziadzio",
    "Jakob D. Kunz",
    "Kaiqu Liang",
    "Alexander Lo",
    "Brian Pulfer",
    "Steven Walton",
    "Charig Yang",
    "Kai Han",
    "Samuel Albanie"
   ],
   "org": "University of Cambridge",
   "url": "https://github.com/jonathan-roberts1/zerobench"
  },
  "released": "2025-02",
  "repo_url": "https://github.com/jonathan-roberts1/zerobench",
  "saturation": {
   "as_of": "",
   "note": "All 20 baselines evaluated at release scored 0% pass@1 and 0% pass^5 (correct on all of 5 samples); the best, Gemini 2 Flash Thinking, reached 5% pass@5 (correct on at least one of 5 samples). Tracking a further 26 models over the following year, the paper reports state-of-the-art climbing to 19% pass@5 (Gemini 3 Pro) and 6% pass^5 (GPT-5.2) -- real but still small progress against a 100-point scale, and the paper notes 47% of the 100 main questions had been answered correctly at least once, by at least one of the 46 evaluated models, cumulatively. The authors state this slower-than-typical erosion, compared to other hard benchmarks tracked over the same period, supports the benchmark's intended longevity.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models (Roberts et al., arXiv:2502.09696)",
    "url": "https://arxiv.org/abs/2502.09696"
   },
   {
    "accessed": "2026-09-08",
    "title": "jonathan-roberts1/zerobench GitHub repository",
    "url": "https://github.com/jonathan-roberts1/zerobench"
   },
   {
    "accessed": "2026-09-08",
    "title": "jonathan-roberts1/zerobench dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/jonathan-roberts1/zerobench"
   },
   {
    "accessed": "2026-09-08",
    "title": "ZeroBench project page and leaderboard",
    "url": "https://zerobench.github.io/"
   }
  ],
  "status": "active",
  "subcategory": "visual reasoning, adversarially filtered to be unsolved at release",
  "summary": "100 hand-made visual reasoning questions filtered so no evaluated frontier model answered any correctly at release, plus 334 subquestions to track partial progress.",
  "tags": [
   "multimodal",
   "visual-reasoning",
   "adversarial-filtering",
   "unsolved-at-release",
   "exact-match"
  ],
  "task_format": "Open-ended, free-text answers, required in the format \"{final_answer}\". Grading is exact string match against the reference answer, with no partial credit on the main questions, since the authors found no distance metric could fairly cover the diversity of correct-answer formats; questions with binary, multiple-choice or small-integer (under 10) answers were deliberately excluded during curation to keep guessing from working.\n"
 }
}