AI2D (AI2 Diagrams)

Multiple-choice question answering over labeled grade-school science diagrams, testing whether a model can connect diagram text, structure and layout to a question.

Also known as: AI2 Diagrams, AI2D-Test

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorymultimodal
Subcategorydiagram question answering
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size3088
Dataset licenceCC BY-SA
PublisherAllen Institute for AI (AI2)

What it measures

Given an annotated science diagram (for example a food web, the water cycle, or a plant cell) and a multiple-choice question about it, the model must read embedded text labels and diagrammatic relationships between parts and choose the correct answer from four options.

Task format

Multiple-choice visual question answering: one diagram image, one question, four answer options, single best choice.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Gemini 2.5 ProGoogle DeepMind95.82026-04
Gemini 2.5 Pro Preview 05-06Google DeepMind95.82026-04
Gemini 2.5 Pro Preview 06-05Google DeepMind95.82026-04
Gemini 2.5 Pro Preview TTSGoogle DeepMind95.82026-04
GPT-4OpenAI95.22026-04
GPT-4.1OpenAI95.22026-04
GPT-4.1 nanoOpenAI95.22026-04
GPT-4oOpenAI94.22026-04
GPT-4o (2024-05-13)OpenAI94.22026-04
GPT-4o (2024-08-06)OpenAI94.22026-04
GPT-4o (2024-11-20)OpenAI94.22026-04
NVLM D 72BNVIDIA94.22026-04
Pixtral Large (latest)Mistral AI93.82026-04
Llama 3.2 90B VisionMeta92.32026-04
Llama 3.2 90B Vision InstructMeta92.32026-04
GPT-4.1 miniOpenAI91.52026-04
GPT-4 TurboOpenAI89.42026-04
Claude Haiku 3Anthropic88.72026-04
Qwen2.5-VL 72B InstructAlibaba / Qwen Team88.72026-04
Claude Opus 3Anthropic88.12026-04
Claude Sonnet 3Anthropic88.12026-04
Qwen3 VL 8B InstructAlibaba / Qwen Team85.72026-04
Gemma 3 27BGoogle DeepMind84.52026-04
Gemma 3 12BGoogle DeepMind84.22026-04
Gemini 2.0 FlashGoogle DeepMind83.32026-04
Qwen2 VL 7B InstructAlibaba / Qwen Team83.02025-03
Qwen2 VL 7B Instruct AWQAlibaba / Qwen Team83.02025-03
MiniCPM V 4OpenBMB82.92026-04
MiniCPM V 4 5OpenBMB82.92026-04
MiniCPM V 4 5 ggufOpenBMB82.92026-04
Gemini 1.5 ProGoogle DeepMind82.52026-04
Claude Sonnet 3.5Anthropic80.22026-04
Claude Sonnet 3.5 v2Anthropic80.22026-04
Pixtral 12BMistral AI79.52026-04
Qwen2.5-VL 7B InstructAlibaba / Qwen Team79.52026-04
Gemini 2.0 Flash LiteGoogle DeepMind78.52026-04
Phi 3.5 vision instructMicrosoft78.12026-04
Gemma 3 4BGoogle DeepMind74.82026-04
gemma 3 4B ptGoogle DeepMind74.82026-04

Data

This page as JSON · Edit on GitHub