{
 "body": "\n## What it measures\n\nChartQA shows a model a chart image, drawn from real published sources, together with a natural-language question about it. The model must give a short answer that may require reading a value directly off the chart, comparing two or more values, or carrying out simple arithmetic or logical reasoning across several data points.\n\nIt targets the gap between simple lookup questions and the compositional, visually grounded questions people actually ask about charts, which most earlier chart-QA datasets did not cover, since those relied on fixed-vocabulary answers and template-based questions.\n\n## How it is scored\n\nThe metric is relaxed accuracy: a numeric answer counts as correct if it falls within 5% of the gold value, and a non-numeric answer must match the reference string exactly. About 9.6K of the roughly 32.7K total question-answer pairs were written directly by human annotators; the rest were generated from human-written chart summaries using a fine-tuned T5 model and then checked. Because the answer space is open-vocabulary rather than fixed, models are typically evaluated end to end rather than as multiple choice, and some reporters split scores by the human-written and machine-generated subsets rather than reporting one pooled number.\n\n## Dataset and licence\n\nThe dataset totals roughly 9.6K human-written and 23.1K machine-generated question-answer pairs over chart images drawn from public chart-hosting sources; the GitHub repository lists Pew Research Center charts among them. Both the GitHub repository and the Hugging Face dataset card list the licence as GPL-3.0. Questions, chart images and gold answers are all public, with an official train, validation and test split; the exact total image count was not confirmed from the sources read for this page.\n\n## Who publishes it\n\nChartQA was introduced by Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty and Enamul Hoque, appearing in Findings of the Association for Computational Linguistics: ACL 2022. The vis-nlp GitHub organisation maintains the reference repository and the Hugging Face dataset mirror. There is no separately maintained leaderboard, so scores mostly appear as one line among many benchmarks in individual model or lab reports.\n\n## Lineage\n\nChartQA does not sit inside a family in this repository's taxonomy and has no direct predecessor; it superseded earlier, more template-based chart-QA datasets by adding compositional and visual-reasoning questions. A harder successor, ChartQAPro, has since been published to address ceiling effects and add more diverse question types; it does not yet have a page in this repository.\n\n## Saturation and contamination\n\nA current ceiling was not independently confirmed against a publisher page during this research pass, so saturation status is recorded as unknown rather than asserted; this repository's own model cards suggest several current vision-language models score in the high 80s to mid 90s on relaxed accuracy, which points toward the benchmark being approached from the top, but that pattern has not been verified here against a primary source. Contamination risk is medium: the dataset, including its training split, has been fully public since 2022, and fine-tuning on the ChartQA training split before reporting a test-split score is standard, expected practice for this benchmark, not an unusual leak, but it does mean a ChartQA score is not a zero-shot measurement for models that did so.\n\n## How to run it\n\nThe reference implementation and evaluation scripts live in vis-nlp/ChartQA, alongside separate files for the human-authored (ChartQA-H) and machine-generated (ChartQA-M) question sets and the underlying data tables behind each chart. No lm-evaluation-harness, inspect_evals, HELM, OpenCompass or BIG-bench task name could be confirmed from those projects' own documentation during this research pass. Reported numbers are hardest to compare when a reporter does not say whether they fine-tuned on the ChartQA training split, evaluated the human-written and machine-generated subsets separately or pooled, and whether they used the 5% relaxed-accuracy tolerance or exact match.\n\n## Reading the numbers\n\nA high ChartQA score shows a model can read values off an image accurately and chain simple arithmetic or comparisons across them, a reasonable proxy for real chart-reading tasks like reading a dashboard or a slide. It does not test chart types, question styles or reasoning depth beyond what is in the original 2022 set, so a very high score does not guarantee robustness on newer, harder chart benchmarks such as ChartQAPro. Check whether a reported number is pooled across the human and machine-generated subsets, since the two are not equally difficult, and whether the model was fine-tuned on ChartQA's own training data before being tested on it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "chartqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 90.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 90.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NVLM D 72B",
   "model_id": "nvidia/nvlm-d-72b",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 86.0,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision",
   "model_id": "meta/llama-3-2-90b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.5,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4",
   "model_id": "openbmb/minicpm-v-4",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 84.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5",
   "model_id": "openbmb/minicpm-v-4-5",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 84.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5 gguf",
   "model_id": "openbmb/minicpm-v-4-5-gguf",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 84.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.0,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 82.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 vision instruct",
   "model_id": "microsoft/phi-3-5-vision-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 81.8,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 81.1,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 80.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 80.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 79.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 76.2,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 75.7,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 74.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 74.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 74.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2",
   "model_id": "xai/grok-2",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Latest",
   "model_id": "xai/grok-2-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision",
   "model_id": "xai/grok-2-vision",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision (1212)",
   "model_id": "xai/grok-2-vision-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision Latest",
   "model_id": "xai/grok-2-vision-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 73.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 72.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 VL 3B Instruct",
   "model_id": "qwen/qwen2-5-vl-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 72.5,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Vision",
   "model_id": "cohere/command-a-vision-07-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 70.5,
   "source": "multimodal-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 66.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 7B Instruct",
   "model_id": "qwen/qwen2-5-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 65.8,
   "source": "multimodal-evals"
  }
 ],
 "page": {
  "aliases": [
   "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning"
  ],
  "category": "multimodal",
  "contamination": {
   "note": "Train and test questions and answers are fully public since 2022, and many vision-language models explicitly fine-tune on the ChartQA training split before reporting a test-split score. That is expected, standard practice for this benchmark rather than an unusual leak, but it means scores are not comparable to a zero-shot evaluation.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "GPL-3.0",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "About 9.6K human-written question-answer pairs plus about 23.1K machine-generated from human-written chart summaries (about 32.7K total, per the paper's abstract). Exact chart-image count was not confirmed from the sources read for this page.",
   "splits": "train / val / test (official split shipped in the repository)",
   "url": "https://huggingface.co/datasets/ahmed-masry/ChartQA"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "chartqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ChartQA shows a model a chart image, drawn from real published sources, together with a natural-language question about it. The model must produce a short answer, which may require reading a value directly off the chart, comparing two or more values, or doing simple arithmetic or logical reasoning across several data points. Roughly three in ten question-answer pairs come from human annotators who wrote compositional, visually grounded questions; the rest were generated from human-written chart summaries and then checked. The task exercises visual perception of chart elements, such as axes, legends and bar heights, together with the reasoning needed to compute an answer, not just lookup.",
  "metric": {
   "baseline_note": "Numeric answers are scored correct within a 5% tolerance of the gold value; non-numeric answers require an exact string match.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "relaxed accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "ChartQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2203.10244",
   "title": "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning",
   "url": "https://arxiv.org/abs/2203.10244",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Ahmed Masry",
    "Do Xuan Long",
    "Jia Qing Tan",
    "Shafiq Joty",
    "Enamul Hoque"
   ],
   "org": "",
   "url": "https://github.com/vis-nlp/ChartQA"
  },
  "released": "2022",
  "repo_url": "https://github.com/vis-nlp/ChartQA",
  "saturation": {
   "as_of": "",
   "note": "Not independently verified against a primary publisher source in this research pass. This repository's own model cards show several current vision-language models reporting scores in the high 80s to mid 90s on relaxed accuracy, which would suggest the benchmark is being approached from the top, but that has not been confirmed here against a primary source.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning",
    "url": "https://arxiv.org/abs/2203.10244"
   },
   {
    "accessed": "2026-09-07",
    "title": "ChartQA (ACL Anthology)",
    "url": "https://aclanthology.org/2022.findings-acl.177/"
   },
   {
    "accessed": "2026-09-07",
    "title": "vis-nlp/ChartQA (GitHub repository)",
    "url": "https://github.com/vis-nlp/ChartQA"
   },
   {
    "accessed": "2026-09-07",
    "title": "ChartQA dataset card (Hugging Face)",
    "url": "https://huggingface.co/datasets/ahmed-masry/ChartQA"
   }
  ],
  "status": "active",
  "subcategory": "chart and figure question answering",
  "summary": "Visual question answering over real-world chart images that require reading values off the chart and doing arithmetic or logical reasoning to answer.",
  "tags": [
   "multimodal",
   "chart-understanding",
   "visual-question-answering",
   "document-understanding"
  ],
  "task_format": "Visual question answering over chart images; open-vocabulary short answers (numbers, phrases, yes/no)"
 }
}