{
 "body": "\n## What it measures\n\nMathVista tests mathematical reasoning that depends on understanding a visual context, rather than\nreasoning over text alone. It spans five task types: figure question answering over charts and plots,\ngeometry problem solving, math word problems embedded in images, textbook question answering, and\ngeneral visual question answering with a quantitative answer. Images range across natural photos,\ngeometry diagrams, abstract and synthetic scenes, function plots and puzzle-test figures, and\nquestions draw on seven reasoning types from arithmetic and algebra through logical, statistical and\nscientific reasoning.\n\nThe benchmark addresses a specific gap: prior work studied mathematical reasoning in text and,\nseparately, visual question answering, but not the intersection, even though many real-world\nmathematical problems (reading a chart, working a geometry diagram, interpreting a textbook figure)\nare inherently visual.\n\n## How it is scored\n\nModels are graded on accuracy: the fraction of examples answered correctly, either by selecting the\nright multiple-choice letter or producing the correct free-form numeric value (an integer, a decimal\nto one or two places, or a list). The paper's evaluation runs in three stages: the model generates a\nfull response, a GPT-4-based extractor pulls the short final answer from it (measured at over 99.5%\naccuracy on a 200-example pilot), and the extracted answer is normalized and scored. The paper reports\na measured random-guess baseline of 17.9% (guessing the most common option or value per question type)\nrather than a fixed rate, since answer format varies by question. A human baseline of 60.3% comes from\nMechanical Turk annotators with at least a high-school education, each completing five testmini\nquestions within 20 minutes.\n\n## Dataset and licence\n\nMathVista pools 6,141 examples from 31 source datasets: 9 existing math-focused QA datasets, 19\nexisting general VQA datasets filtered and hand-verified for mathematical content, and three datasets\nbuilt for this paper (IQTest, puzzle-figure logical reasoning; FunctionQA, algebraic reasoning over\nfunction plots; PaperQA, scientific reasoning over academic figures), contributing 736 new examples.\nThe data splits into a 1,000-example \"testmini\" subset, sampled to match the full set's distribution\nand used for most reported scores, and a 5,141-example \"test\" subset whose answers are withheld to\nlimit contamination. It is distributed on Hugging Face (`AI4Math/MathVista`) under CC BY-SA 4.0 per\nthat dataset card; the arXiv paper's own front matter separately states CC BY 4.0.\n\n## Who publishes it\n\nMathVista was introduced by Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh\nHajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley and Jianfeng Gao, a collaboration spanning UCLA,\nthe University of Washington and Microsoft Research, posted to arXiv in October 2023 and accepted at\nICLR 2024. The authors maintain the project site and leaderboard at mathvista.github.io, which takes\nsubmissions by email and publishes separate testmini and test leaderboards.\n\n## Lineage\n\nMathVista has no single named predecessor; it aggregates 28 pre-existing math and VQA datasets plus\nthree newly built ones, rather than extending any one prior benchmark. It has no formally named\nsuccessor and no variant is tracked in this repository yet, though it is often cited alongside later,\nharder visual-math benchmarks as scores on it have risen.\n\n## Saturation and contamination\n\nAt release, the best-performing model (GPT-4V) scored 49.9%, a 15.1-point jump over the next-best\nmodel (Multimodal Bard) but still 10.4 points short of the 60.3% human baseline. The project's own\nleaderboard, fetched directly for this page, lists its top testmini entry as \"DreamPRM (o4-mini)\" at\n85.2%, dated June 2025, well clear of the human baseline, with no newer entry recorded as of this\nresearch \u2014 a roughly 35-point gain in about twenty months, pointing toward saturation, though this\nsnapshot may lag the true state of the art. Contamination risk sits at medium: testmini has been\nfully public with released answers for close to two years and is the split nearly everyone reports,\neven though a larger, answer-withheld test subset exists to guard against this.\n\n## How to run it\n\nUK AISI's `inspect_evals` package implements this as the `mathvista` task over the public `testmini`\nsplit, but scores answers with a regular-expression match against a fixed answer-format instruction\nrather than the paper's own GPT-4-based extractor \u2014 a protocol difference that can affect\ncomparability, particularly for free-form numeric answers. No lm-evaluation-harness, HELM,\nOpenCompass or BIG-bench task was confirmed for this benchmark. Because most published numbers report\ntestmini rather than the larger held-out test subset, and extraction method differs by harness, treat\nscores from different sources as only roughly comparable.\n\n## Reading the numbers\n\nA strong MathVista score indicates a model can combine visual perception with quantitative reasoning:\nreading a value off a chart, tracking a geometric relationship, or extracting a number from a diagram\nand computing with it correctly. It is a composite of five task types, so one overall number can hide\nlarge gaps between, say, geometry problem solving and visual question answering \u2014 read the per-task\nbreakdown alongside the headline score. Given how far top scores have moved past the human baseline,\nand how long testmini has circulated publicly, treat a very high recent score with some contamination\ncaution, and check it against the harder, answer-withheld test subset where available.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mathvista",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.7,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 73.7,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 70.7,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 70.7,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 70.7,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 68.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 67.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 67.7,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 67.7,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4",
   "model_id": "openbmb/minicpm-v-4",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 66.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5",
   "model_id": "openbmb/minicpm-v-4-5",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 66.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5 gguf",
   "model_id": "openbmb/minicpm-v-4-5-gguf",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 66.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 65.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 65.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NVLM D 72B",
   "model_id": "nvidia/nvlm-d-72b",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 65.2,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 63.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 63.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 63.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 63.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 63.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.9,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 62.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 62.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 62.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.2,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.2,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 58.2,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision",
   "model_id": "meta/llama-3-2-90b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 57.3,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 57.3,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 55.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 54.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 54.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 54.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 54.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 54.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 54.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 52.5,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision",
   "model_id": "meta/llama-3-2-11b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 51.5,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision Instruct",
   "model_id": "meta/llama-3-2-11b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 51.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 50.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 50.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 50.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 50.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 50.0,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2",
   "model_id": "xai/grok-2",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 48.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Latest",
   "model_id": "xai/grok-2-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 48.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision",
   "model_id": "xai/grok-2-vision",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 48.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision (1212)",
   "model_id": "xai/grok-2-vision-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 48.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision Latest",
   "model_id": "xai/grok-2-vision-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 48.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 48.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 47.9,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Vision",
   "model_id": "cohere/command-a-vision-07-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 46.5,
   "source": "multimodal-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 VL 3B Instruct",
   "model_id": "qwen/qwen2-5-vl-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 45.8,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3.5 vision instruct",
   "model_id": "microsoft/phi-3-5-vision-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 43.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 43.8,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 7B Instruct",
   "model_id": "qwen/qwen2-5-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 42.5,
   "source": "multimodal-evals"
  }
 ],
 "page": {
  "aliases": [],
  "category": "math",
  "contamination": {
   "note": "The 1,000-example testmini subset is fully public with released answers and has been the standard reported split for nearly two years, making it a plausible target for training-data inclusion. The larger test subset (5,141 examples) has its answers withheld specifically to limit this, with submissions handled through an online evaluation platform, but most published leaderboard numbers are testmini scores rather than test scores.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en",
    "zh",
    "fa"
   ],
   "license": "CC BY-SA 4.0",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": true,
   "size": 6141,
   "size_note": "6,141 examples total, split into a public \"testmini\" subset of 1,000 (used for most reported leaderboard numbers and for the human baseline) and a \"test\" subset of the remaining 5,141, whose answers are not publicly released to limit contamination. The set pools 9 math-focused and 19 general VQA datasets (31 source datasets total, 28 pre-existing) plus three datasets built for this paper (IQTest, FunctionQA, PaperQA); 736 examples are newly curated.\n",
   "splits": "testmini (1,000, public); test (5,141, answers withheld)",
   "url": "https://huggingface.co/datasets/AI4Math/MathVista"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice L",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "mathvista",
   "lm_eval": "",
   "opencompass": "",
   "other": "The original paper's own evaluation used a three-stage protocol (response generation, then a GPT-4-based answer extractor measured at over 99.5% accuracy on a 200-example pilot, then normalization and scoring) rather than exact string matching. UK AISI's inspect_evals implementation instead extracts answers with a regular expression against a fixed answer-format instruction, which is a real protocol difference from the paper's own GPT-4 extractor and can affect comparability of reported scores.\n"
  },
  "id": "mathvista",
  "last_updated": "2025-06",
  "leaderboard_url": "https://mathvista.github.io/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "MathVista tests whether a model can perform mathematical reasoning that depends on understanding a visual context, rather than reasoning over text alone. It spans five task types: figure question answering (reasoning over charts and plots), geometry problem solving, math word problems set in images, textbook question answering, and general visual question answering with a numeric or quantitative answer. Underlying visual contexts include natural images, geometry diagrams, abstract and synthetic scenes, function plots, and puzzle-test figures, and the questions draw on seven reasoning types from arithmetic and algebra to logical, statistical and scientific reasoning.\n",
  "metric": {
   "baseline_note": "17.9% is the paper's own measured random-guess baseline (choosing the most frequent option or value by question type), not a fixed theoretical rate, since the benchmark mixes multiple-choice questions of varying option counts with free-form numeric and list answers. The 60.3% human baseline comes from Amazon Mechanical Turk annotators with a high-school diploma or higher, each answering five questions from the testmini subset within 20 minutes.\n",
   "direction": "higher_is_better",
   "human_baseline": 60.3,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 17.9,
   "unit": "%"
  },
  "name": "MathVista",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2310.02255",
   "title": "MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts",
   "url": "https://arxiv.org/abs/2310.02255",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Pan Lu",
    "Hritik Bansal",
    "Tony Xia",
    "Jiacheng Liu",
    "Chunyuan Li",
    "Hannaneh Hajishirzi",
    "Hao Cheng",
    "Kai-Wei Chang",
    "Michel Galley",
    "Jianfeng Gao"
   ],
   "org": "UCLA, University of Washington, Microsoft Research (collaboration)",
   "url": "https://mathvista.github.io/"
  },
  "released": "2023-10",
  "repo_url": "https://github.com/lupantech/MathVista",
  "saturation": {
   "as_of": "2025-06",
   "note": "At release (October 2023), the best model was GPT-4V at 49.9%, itself 10.4 points short of the 60.3% human baseline. The project's own leaderboard, fetched directly for this page, currently lists its top testmini entry as \"DreamPRM (o4-mini)\" at 85.2% (dated 2025-06-04, citing arXiv:2505.20241), well above the human baseline and with no newer entry recorded since. Top scores are now comfortably clear of the original human baseline, though the leaderboard's own data file shows no update after mid-2025, so a more recent snapshot may exist elsewhere.\n",
   "status": "watch",
   "top_score": 85.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (arXiv:2310.02255)",
    "url": "https://arxiv.org/abs/2310.02255"
   },
   {
    "accessed": "2026-09-08",
    "title": "AI4Math/MathVista dataset card",
    "url": "https://huggingface.co/datasets/AI4Math/MathVista"
   },
   {
    "accessed": "2026-09-08",
    "title": "MathVista project page and leaderboard",
    "url": "https://mathvista.github.io/"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals: mathvista task implementation",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mathvista"
   }
  ],
  "status": "active",
  "subcategory": "visual mathematical reasoning",
  "summary": "6,141 examples testing mathematical reasoning across charts, diagrams, word problems and textbook figures, pooled from 28 existing datasets plus three new ones.",
  "tags": [
   "math",
   "multimodal",
   "multiple-choice",
   "visual-reasoning",
   "benchmark-aggregation"
  ],
  "task_format": "An image paired with a question, answered as either a multiple-choice letter or a free-form numerical answer (integer, one or two decimal places, or a list), in English.\n"
 }
}