{
 "body": "\n## What it measures\nAI2D tests whether a model can read a labeled science diagram \u2014 the kind found in a grade-school textbook \u2014 well enough to answer a multiple-choice question about it. A model is shown a diagram image (a food web, the water cycle, a plant cell, a phase-of-the-moon chart) and a question with several answer options, and must choose the correct one. Answering usually requires reading small embedded text labels, tracing arrows and lines between diagram parts, and connecting that structure to the question, rather than just describing the picture. It is an English-language, image-plus-text task.\n\nThe dataset comes from the 2016 paper \"A Diagram Is Worth A Dozen Images,\" which also introduced Diagram Parse Graphs (DPGs), a structured representation of a diagram's constituents and their relationships, and a DPG-based question-answering model. Most modern benchmark reporting only uses the multiple-choice QA task, not the DPG parsing task the paper also proposed.\n\n## How it is scored\nAccuracy: the percentage of questions answered with the correct option out of four. There is no partial credit and no separately weighted \"hard\" subset in the version most benchmark suites report. Evaluation is typically zero-shot: the model sees the diagram and question once and picks an answer, with no worked examples given. Harnesses can differ in whether they present the question and options as a text prompt alongside the raw diagram image or embed the text into the image itself, which is a known source of score differences between reporters.\n\n## Dataset and licence\nThe original release documents \"over 5,000 diagrams and 15,000 questions and answers,\" each diagram annotated with segmented diagram elements, their relationships to each other, and their relationships to the diagram canvas. The version most vision-language model papers evaluate against is a fixed test split of 3,088 question-image pairs (hosted, among other places, as lmms-lab/ai2d on Hugging Face), each with four answer choices. The dataset is released under a CC BY-SA licence. Images were collected by searching Google Images with terms drawn from grade 1-6 science textbook chapter titles, then annotated through Amazon Mechanical Turk.\n\n## Who publishes it\nAI2D was built and released by the Allen Institute for AI (AI2), authored by Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi and Ali Farhadi. It was published on arXiv as arXiv:1603.07396 in 2016. AI2 hosts the raw data and test IDs on its PRIOR project page and through the AWS Open Data registry; there is no single publisher-run leaderboard, so current standings come from whichever vision-language model technical report or third-party tracker chooses to report a score.\n\n## Lineage\nAI2D predates the current wave of instruction-tuned vision-language models and was designed alongside the DPG-parsing and diagram-QA models of its era. A related academic corpus, AI2D-RST, later added rhetorical-structure annotations to a subset of AI2D's diagrams, but it is not tracked as a separate id in this repository. AI2D itself has no tools/agentic variant in this repository.\n\n## Saturation and contamination\nAI2D has been public since 2016 and is mirrored on Hugging Face, Kaggle and AWS, so its images and answer keys are plausibly present in large web-scale pretraining corpora; contamination risk should be treated as at least medium for models trained on broad web crawls. There is no AI2-run leaderboard to check against a definitive current ceiling, but third-party trackers generally show leading vision-language models clustering in the low-to-mid 90s (percent) with a narrow spread between them, consistent with the benchmark separating models less clearly than it once did.\n\n## How to run it\nMost vision-language model evaluation harnesses that include AI2D (for example lmms-eval and VLMEvalKit) load the 3,088-question test split and score plain accuracy over the four options. Check a given harness's task definition for whether it embeds the question text into the image or passes it as a separate text prompt, since scores are not always comparable across that choice.\n\n## Reading the numbers\nA high AI2D score shows a model can align text labels with diagram structure and follow simple diagrammatic relationships \u2014 a proxy for basic scientific-diagram literacy rather than general chart or document understanding. Because the test set is small, public and years old, a top score today says less about frontier capability than it did in 2016; look at harder, more recent chart and diagram benchmarks (for example CharXiv) alongside it, and treat a near-ceiling AI2D score as a baseline expectation rather than a differentiator.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ai2d",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 95.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NVLM D 72B",
   "model_id": "nvidia/nvlm-d-72b",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 94.2,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 93.8,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision",
   "model_id": "meta/llama-3-2-90b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 92.3,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 92.3,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.4,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.7,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 88.7,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.1,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.1,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL 8B Instruct",
   "model_id": "qwen/qwen3-vl-8b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 85.7,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 84.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 83.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4",
   "model_id": "openbmb/minicpm-v-4",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 82.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5",
   "model_id": "openbmb/minicpm-v-4-5",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 82.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5 gguf",
   "model_id": "openbmb/minicpm-v-4-5-gguf",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 82.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 79.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 7B Instruct",
   "model_id": "qwen/qwen2-5-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 79.5,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 vision instruct",
   "model_id": "microsoft/phi-3-5-vision-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 78.1,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  }
 ],
 "page": {
  "aliases": [
   "AI2 Diagrams",
   "AI2D-Test"
  ],
  "category": "multimodal",
  "contamination": {
   "note": "The dataset and its answer keys have been fully public since 2016 and are widely mirrored (Hugging Face, AWS Open Data, Kaggle), so they plausibly appear in large web-scale pretraining corpora.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": true,
   "size": 3088,
   "size_note": "Standard test split (3,088 question-image pairs, four options each) used by most vision-language model evaluation harnesses, hosted as lmms-lab/ai2d on Hugging Face; the original release documents over 5,000 diagrams and 15,000 questions and answers across all splits.",
   "splits": "train / val / test in the original release; a 3,088-item test split is what most modern harnesses score",
   "url": "https://prior.allenai.org/projects/diagram-understanding"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice E",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "ai2d",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Given an annotated science diagram (for example a food web, the water cycle, or a plant cell) and a multiple-choice question about it, the model must read embedded text labels and diagrammatic relationships between parts and choose the correct answer from four options.",
  "metric": {
   "baseline_note": "",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "AI2D (AI2 Diagrams)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1603.07396",
   "title": "A Diagram Is Worth A Dozen Images",
   "url": "https://arxiv.org/abs/1603.07396",
   "year": 2016
  },
  "publisher": {
   "authors": [
    "Aniruddha Kembhavi",
    "Mike Salvato",
    "Eric Kolve",
    "Minjoon Seo",
    "Hannaneh Hajishirzi",
    "Ali Farhadi"
   ],
   "org": "Allen Institute for AI (AI2)",
   "url": "https://allenai.org"
  },
  "released": "2016",
  "repo_url": "",
  "saturation": {
   "as_of": "",
   "note": "Third-party trackers show leading vision-language models clustering in the low-to-mid 90s (percent), with several recent releases separated by well under a point, but AI2D has no publisher-maintained leaderboard to confirm a single current ceiling.",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "A Diagram Is Worth A Dozen Images",
    "url": "https://arxiv.org/abs/1603.07396"
   },
   {
    "accessed": "2026-09-07",
    "title": "lmms-lab/ai2d",
    "url": "https://huggingface.co/datasets/lmms-lab/ai2d"
   },
   {
    "accessed": "2026-09-07",
    "title": "AI2 Diagram Dataset (AI2D) - Registry of Open Data on AWS",
    "url": "https://registry.opendata.aws/allenai-diagrams/"
   },
   {
    "accessed": "2026-09-07",
    "title": "Diagram Understanding - AI2 PRIOR",
    "url": "https://prior.allenai.org/projects/diagram-understanding"
   }
  ],
  "status": "active",
  "subcategory": "diagram question answering",
  "summary": "Multiple-choice question answering over labeled grade-school science diagrams, testing whether a model can connect diagram text, structure and layout to a question.",
  "tags": [
   "diagram-understanding",
   "visual-question-answering",
   "science"
  ],
  "task_format": "Multiple-choice visual question answering: one diagram image, one question, four answer options, single best choice."
 }
}