{
 "body": "\n## What it measures\n\nFigQA is one of seven task categories in LAB-Bench, a suite FutureHouse built to test practical\nbiology-research skills \u2014 literature search, figure and table interpretation, database navigation,\nprotocol understanding, and DNA/protein sequence manipulation \u2014 rather than textbook-style recall. The\ntask shows a model an image of a figure taken from a biology research paper, deliberately stripped of\nits caption, surrounding text and paper title, and asks a multiple-choice question that usually\nrequires integrating several elements of the figure at once. Its authors compare it to a visual\nversion of multi-hop text benchmarks like HotpotQA. FigQA requires the model to be multi-modal but,\nunlike some other LAB-Bench tasks, was designed to need no external tool use.\n\n## How it is scored\n\nEvery question offers an explicit \"insufficient information\" option alongside the substantive\nchoices, so LAB-Bench reports two headline numbers: accuracy (correct answers divided by all\nquestions) and precision (correct answers divided only by questions the model chose to attempt).\nCoverage \u2014 the attempted share \u2014 is reported alongside both, since models vary widely in how often\nthey decline. The paper's own results show this matters: some models achieve reasonable precision by\nattempting few, easy-seeming questions, while others attempt nearly everything and post lower\nprecision as a result. A supplemental open-answer variant (removing the multiple-choice options)\nshowed materially lower scores for the two models tested, indicating some multiple-choice performance\ncomes from eliminating implausible distractors rather than true comprehension.\n\n## Dataset and licence\n\nFigQA contributes 226 of LAB-Bench's more than 2,400 total questions, alongside LitQA2 (248), SuppQA\n(102), TableQA (305), DbQA (650 across 10 subtasks), ProtocolQA (135), SeqQA (750 across 15 subtasks)\nand CloningScenarios (41). Questions were generated manually by the paper's authors and contracted\nbiology experts, who selected papers, extracted a figure, and wrote a question that couldn't be\nanswered from the figure's caption or surrounding text alone. The dataset is released under CC BY-SA\n4.0 and hosted on Hugging Face (`futurehouse/lab-bench`); FutureHouse publishes roughly 80% of each\nsubtask's questions and withholds the remainder to monitor for contamination going forward.\n\n## Who publishes it\n\nLAB-Bench comes from FutureHouse, a nonprofit AI-for-science research lab, with authors Jon M.\nLaurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth\nNarayanan, Manvitha Ponnapati, Andrew D. White and Samuel G. Rodriques. The paper was submitted to\nNeurIPS 2024's Datasets and Benchmarks track and posted to arXiv in July 2024. FutureHouse maintains\nthe reference dataset and states an intent to keep expanding LAB-Bench over time; there is no\nseparate, continuously updated public leaderboard beyond the dataset repository itself.\n\n## Lineage\n\nFigQA has no named predecessor; it is one of eight sibling task categories introduced together in the\noriginal LAB-Bench paper, several of which (LitQA2, SuppQA, DbQA, SeqQA) are explicitly tool-dependent\nby design in a way FigQA is not. This repository tracks one direct variant, `lab_bench_figqa_tools`,\ncovering FigQA scores produced when a model is given tool access such as an image-cropping tool,\nrather than the untooled default this page describes.\n\n## Saturation and contamination\n\nAt release, LAB-Bench's authors found FigQA the hardest category in the suite: most models scored\nnear-random precision, with Claude 3.5 Sonnet a clear outlier above the rest. Progress since has been\nuneven rather than steadily upward \u2014 Anthropic's Claude 4 System Card (May 2025) found Claude Opus 4\nand Claude Sonnet 4 scoring *below* Claude Sonnet 3.7 on this specific task, even as those models\nimproved on other LAB-Bench categories. The most recent figure located in this research, Claude Opus\n4.5's untooled baseline of 54.9% (Anthropic, November 2025), still sits well under the roughly 75%\nhuman baseline plotted in the same source. Contamination risk is comparatively low: FutureHouse\nwithholds a private portion of each subtask specifically to monitor for it, and FigQA's images are not\na single widely-mirrored file that would circulate easily.\n\n## How to run it\n\nUK AISI's `inspect_evals` package implements this task under the exact id `lab_bench_figqa`, pulling\nfrom the same public Hugging Face subset. The original paper's own evaluation ran every model without\nexternal tools and used k-shot prompting; Anthropic's system cards have since evaluated it both this\nway and with tool access, using 0-shot prompting in the more recent case, so scores from different\nsources may not share a prompting protocol even when both report \"no tools.\"\n\n## Reading the numbers\n\nA strong FigQA score suggests a model can extract and cross-reference quantitative or structural\ninformation from an unlabelled scientific figure, a narrower and more visually demanding skill than\ngeneral chart or document QA. It does not by itself indicate broader biology-research competence \u2014\nthat is closer to what LAB-Bench's other, more tool-dependent categories test. Because coverage\n(willingness to attempt a question) varies so much between models, always check whether a reported\nnumber is accuracy or precision before comparing two models, and treat FigQA scores from before and\nafter a change in tool access or prompting protocol as not directly comparable.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "lab_bench_figqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 79.7,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 58.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  }
 ],
 "page": {
  "aliases": [
   "FigQA"
  ],
  "category": "domain",
  "contamination": {
   "note": "FutureHouse withholds roughly 20% of each subtask, including FigQA, specifically to monitor for contamination, and the public/private split performance is reported separately in the paper's supplemental material. Because FigQA questions are built from arbitrary figures the authors screenshotted from papers rather than from a fixed, widely mirrored public file, the paper's own limitations discussion does not flag it as a high leakage risk the way heavily-circulated benchmarks are.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA 4.0",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": true,
   "size": 226,
   "size_note": "226 questions, one of eight rows in the paper's category table (LitQA2 248, SuppQA 102, FigQA 226, TableQA 305, DbQA 650 across 10 subtasks, ProtocolQA 135, SeqQA 750 across 15 subtasks, CloningScenarios 41), for a combined LAB-Bench total of over 2,400 questions. FutureHouse makes roughly 80% of each subtask public and holds back the rest to monitor for future contamination.\n",
   "splits": "public subset released on Hugging Face; a private held-out portion is not released",
   "url": "https://huggingface.co/datasets/futurehouse/lab-bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice L",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "lab_bench_figqa",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "lab_bench_figqa",
  "last_updated": "2025-09",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "lab_bench_figqa_tools"
   ]
  },
  "measures": "FigQA is one of seven task categories in FutureHouse's LAB-Bench, a suite built to test practical biology-research skills rather than textbook recall. FigQA shows the model an image of a figure taken from a biology research paper, with no caption, surrounding text or paper title, and asks a multiple-choice question that usually requires integrating information from several elements of the figure at once. The task's authors describe it as a visual analogue of multi-hop text benchmarks such as HotpotQA. It requires a model to be multi-modal but was designed, by its authors, to need no external tool use: the image alone should contain everything needed to answer.\n",
  "metric": {
   "baseline_note": "LAB-Bench reports two scores throughout: accuracy (correct / all questions) and precision (correct / attempted, i.e. excluding declined questions); coverage (attempted / all) is reported separately. Because models decline at very different rates, precision and accuracy can diverge substantially for the same model, and both are worth reading rather than either alone. The original paper gives no single printed human-baseline percentage for FigQA (human performance is shown only as a line on a bar chart); Anthropic's Claude 4 System Card (May 2025) plots a FigQA human baseline at roughly three-quarters correct, well above every model shown in that chart.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "precision (correct / attempted)",
   "unit": "%"
  },
  "name": "LAB-Bench: FigQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2407.10362",
   "title": "LAB-Bench: Measuring Capabilities of Language Models for Biology Research",
   "url": "https://arxiv.org/abs/2407.10362",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Jon M. Laurent",
    "Joseph D. Janizek",
    "Michael Ruzo",
    "Michaela M. Hinks",
    "Michael J. Hammerling",
    "Siddharth Narayanan",
    "Manvitha Ponnapati",
    "Andrew D. White",
    "Samuel G. Rodriques"
   ],
   "org": "FutureHouse",
   "url": "https://github.com/Future-House/LAB-Bench"
  },
  "released": "2024-07",
  "repo_url": "https://github.com/Future-House/LAB-Bench",
  "saturation": {
   "as_of": "2025-11",
   "note": "At the July 2024 release, all models the authors tested showed \"near-random\" FigQA precision except Claude 3.5 Sonnet, which scored well above the rest. Anthropic's Claude 4 System Card (May 2025) later found Claude Opus 4 and Claude Sonnet 4 scoring lower than Claude Sonnet 3.7 had on this specific task even as they improved elsewhere. The most recent figure found in this research, Claude Opus 4.5's no-tools, no-reasoning baseline of 54.9% (Anthropic, November 2025), remains well below the roughly 75% human baseline shown in the same source, so the task is not saturated.\n",
   "status": "open",
   "top_score": 54.9
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "LAB-Bench: Measuring Capabilities of Language Models for Biology Research (arXiv:2407.10362)",
    "url": "https://arxiv.org/abs/2407.10362"
   },
   {
    "accessed": "2026-09-08",
    "title": "Future-House/LAB-Bench (repository description and licence)",
    "url": "https://github.com/Future-House/LAB-Bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "futurehouse/lab-bench dataset card",
    "url": "https://huggingface.co/datasets/futurehouse/lab-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "System Card: Claude Opus 4 & Claude Sonnet 4 (Anthropic, May 2025), Section 7.2.4.7",
    "url": "https://www.anthropic.com/claude-4-system-card"
   },
   {
    "accessed": "2026-09-08",
    "title": "System Card: Claude Opus 4.5 (Anthropic, November 2025), Section 2.21",
    "url": "https://www.anthropic.com/claude-opus-4-5-system-card"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals: lab_bench_figqa task implementation",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/lab_bench"
   }
  ],
  "status": "active",
  "subcategory": "biology research - figure interpretation",
  "summary": "226 multiple-choice questions that give a model only a biology-paper figure image, no caption or text, and ask it to reason about the figure's content.",
  "tags": [
   "domain",
   "biology",
   "multimodal",
   "multiple-choice",
   "figure-interpretation"
  ],
  "task_format": "Multiple-choice question over a single figure image (no caption or paper text provided), five or more options including an explicit option to decline for lack of information.\n"
 }
}