{
 "body": "\n## What it measures\n\nDocVQA asks a model to read a scanned or born-digital document image \u2014 a letter, memo, form, or\nreport \u2014 and answer a short question about its content in free text. Questions probe layout-dependent\nreading: totals in tables, handwritten notes, form field values, figure captions and running text, not\njust plain paragraphs. It is a single-image, single-turn, English-language task that combines OCR,\nlayout understanding and reading comprehension in one model call.\n\nThe images come from real historical business documents rather than clean scans, so a model has to\ncope with skew, low contrast, typewriter fonts and handwriting alongside typeset text.\n\n## How it is scored\n\nDocVQA is scored with ANLS, Average Normalized Levenshtein Similarity, a metric originally proposed for\nthe ST-VQA benchmark: it compares a predicted string against reference answers using normalized edit\ndistance, so answers that are close but not exact (typical of OCR noise) still earn partial credit\ninstead of zero. Each question can have more than one accepted reference answer, and the best match\ncounts. The paper also reports plain exact-match accuracy as a secondary metric, but treats ANLS as\nprimary specifically because it tolerates minor OCR-style mismatches. There is no fixed random baseline\nbecause answers are open-ended text rather than multiple choice. Because reference answers on the\nofficial test split are withheld by the organizers, most published leaderboard numbers today evaluate on\nthe validation split instead.\n\n## Dataset and licence\n\n50,000 questions were collected over 12,767 document images sourced from the UCSF Industry Documents\nLibrary, a public archive of internal tobacco, drug and chemical industry records; images are split\nroughly 80/10/10 into train, validation and test. Questions and answers were crowdsourced against each\nimage, with multiple accepted reference answers per question where wording could reasonably vary.\nNeither docvqa.org nor the paper states an explicit licence for the released question-answer data\nitself; a Hugging Face mirror maintained by lmms-lab tags the packaged dataset Apache-2.0, but that is\nthe mirror's own applied tag rather than a licence set by DocVQA's organizers, so this page leaves the\nlicence field unset rather than guess.\n\n## Who publishes it\n\nDocVQA was introduced by Minesh Mathew, Dimosthenis Karatzas, R. Manmatha and C.V. Jawahar, first posted\nto arXiv in July 2020 and published at WACV 2021. It runs as Challenge 17 of the Robust Reading\nCompetition, organized jointly by the Computer Vision Center at the Universitat Aut\u00f2noma de Barcelona\nand IIIT Hyderabad, with Amazon's involvement through co-author R. Manmatha. The docvqa.org site also\ncredits Rub\u00e8n P\u00e9rez Tito, Ernest Valveny and Artemis Llabr\u00e9s, who run related document-QA challenges\n(InfographicVQA, DocCVQA) under the same portal. The organizers continue to run the Robust Reading\nCompetition site that hosts the official leaderboard.\n\n## Lineage\n\nDocVQA is the base task in a small family of document-image QA challenges run by the same group:\nInfographicVQA asks questions over infographics rather than plain documents, DocCVQA is a conversational\nvariant, and MP-DocVQA (Multipage DocVQA) extends questions across multi-page documents instead of a\nsingle page. None of those has its own page in this repository yet. DocVQA itself names no predecessor;\nit was among the first large VQA-style datasets built specifically around scanned documents rather than\nnatural images.\n\n## Saturation and contamination\n\nLeading vision-language models now sit close to the paper's own human ANLS baseline of 0.981 on the\ntest split: llm-stats.com's DocVQA leaderboard put Qwen2.5 VL 72B Instruct at 0.964 as of early\nSeptember 2026, a gap of under two points. That leaves little room before the metric's ceiling of 1.0\nor the original human reference point. The validation split's questions and reference answers have been\npublic since 2020, so six years of web presence make some contamination of that split plausible even\nthough the official test-split answers remain withheld; risk sits at medium rather than high because\nthose specific held-out answers were never published.\n\n## How to run it\n\nThe reference implementation and official leaderboard live at docvqa.org's Robust Reading Competition\nportal, which accepts predictions against the withheld test split. Inspect Evals (UK AISI's\ninspect_evals package) ships a `docvqa` task that evaluates the public validation split with the same\nANLS metric, which is the practical way most model developers reproduce a comparable number without a\ncompetition submission. Because the test split is closed, most reported scores in practice come from\nthe validation split, and papers do not always say which split, prompt format or image preprocessing\n(resolution, cropping) they used, which makes cross-paper comparisons somewhat noisy.\n\n## Reading the numbers\n\nA high DocVQA score shows a model can read structured business documents and locate the right field or\nfigure, which transfers reasonably well to real invoice, form and report understanding. It says much\nless about multi-page reasoning, arithmetic over table values, or documents in languages other than\nEnglish, none of which this benchmark covers. Given how close current models sit to the original human\nbaseline, a strong DocVQA number today mostly confirms basic competence rather than separating frontier\nmodels from each other; for that, pair it with a harder, less saturated document benchmark such as\nInfographicVQA or a multi-page task.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "docvqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL 32B Instruct",
   "model_id": "qwen/qwen3-vl-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 96.5,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL 8B Instruct",
   "model_id": "qwen/qwen3-vl-8b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 96.1,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct",
   "model_id": "qwen/qwen2-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 94.5,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2025-03",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 VL 7B Instruct AWQ",
   "model_id": "qwen/qwen2-vl-7b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 94.5,
   "source": "open-llm-leaderboard-v2, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 94.4,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 94.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 93.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 93.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4",
   "model_id": "openbmb/minicpm-v-4",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 92.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5",
   "model_id": "openbmb/minicpm-v-4-5",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 92.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5 gguf",
   "model_id": "openbmb/minicpm-v-4-5-gguf",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 92.9,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NVLM D 72B",
   "model_id": "nvidia/nvlm-d-72b",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 92.6,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision",
   "model_id": "meta/llama-3-2-90b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.1,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.1,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 90.1,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision",
   "model_id": "meta/llama-3-2-11b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.4,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision Instruct",
   "model_id": "meta/llama-3-2-11b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.4,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 12B",
   "model_id": "google/gemma-3-12b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.1,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 85.5,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 VL 3B Instruct",
   "model_id": "qwen/qwen2-5-vl-3b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 85.2,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 83.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 83.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2",
   "model_id": "xai/grok-2",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Latest",
   "model_id": "xai/grok-2-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision",
   "model_id": "xai/grok-2-vision",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision (1212)",
   "model_id": "xai/grok-2-vision-1212",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Grok 2 Vision Latest",
   "model_id": "xai/grok-2-vision-latest",
   "provider": "xai",
   "provider_display": "xAI",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 27B",
   "model_id": "google/gemma-3-27b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command A Vision",
   "model_id": "cohere/command-a-vision-07-2025",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 81.2,
   "source": "multimodal-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.8,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 7B Instruct",
   "model_id": "qwen/qwen2-5-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 78.5,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 3 4B",
   "model_id": "google/gemma-3-4b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 75.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 3 4B pt",
   "model_id": "google/gemma-3-4b-pt",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 75.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals"
  }
 ],
 "page": {
  "aliases": [
   "Document Visual Question Answering"
  ],
  "category": "multimodal",
  "contamination": {
   "note": "The validation split's questions and reference answers have been public since 2020, so six years of web presence make some contamination plausible even though the official test-split answers remain withheld by the organizers.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": false,
   "size": 50000,
   "size_note": "50,000 questions over 12,767 document images, split roughly 80/10/10 into train, validation and test.",
   "splits": "train / validation / test; reference answers on the official test split are withheld by the organizers",
   "url": "https://www.docvqa.org/"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "docvqa",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "docvqa",
  "last_updated": "",
  "leaderboard_url": "https://www.docvqa.org/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "DocVQA gives a model an image of a real document \u2014 a letter, memo, form, table or report \u2014 plus a natural-language question about its content, and asks for a short free-text answer. Questions require reading text in context (totals in a table, a form field's value, a handwritten note, a figure label) rather than isolated OCR, so the task combines text recognition, layout understanding and reading comprehension in a single English-language, single-image, single-turn call.\n",
  "metric": {
   "baseline_note": "Human ANLS measured by the paper's authors on the test split; human exact-match accuracy on the same split was reported separately as 94.36%.\n",
   "direction": "higher_is_better",
   "human_baseline": 0.981,
   "max_score": 1.0,
   "name": "ANLS (Average Normalized Levenshtein Similarity)",
   "random_baseline": null,
   "unit": "score (0-1)"
  },
  "name": "DocVQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2007.00398",
   "title": "DocVQA: A Dataset for VQA on Document Images",
   "url": "https://arxiv.org/abs/2007.00398",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Minesh Mathew",
    "Dimosthenis Karatzas",
    "R. Manmatha",
    "C.V. Jawahar"
   ],
   "org": "Computer Vision Center (Universitat Aut\u00f2noma de Barcelona) and IIIT Hyderabad, with Amazon",
   "url": "https://www.docvqa.org/"
  },
  "released": "2020-07",
  "repo_url": "",
  "saturation": {
   "as_of": "2026-09",
   "note": "llm-stats.com's DocVQA leaderboard put Qwen2.5 VL 72B Instruct at 0.964 ANLS as of early September 2026, within two points of the paper's own human ANLS baseline of 0.981 and close to the metric's ceiling of 1.0.\n",
   "status": "saturated",
   "top_score": 0.964
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "DocVQA: A Dataset for VQA on Document Images",
    "url": "https://arxiv.org/abs/2007.00398"
   },
   {
    "accessed": "2026-09-07",
    "title": "DocVQA: A Dataset for VQA on Document Images (HTML, evaluation metrics and results tables)",
    "url": "https://arxiv.org/html/2007.00398"
   },
   {
    "accessed": "2026-09-07",
    "title": "DocVQA",
    "url": "https://www.docvqa.org/"
   },
   {
    "accessed": "2026-09-07",
    "title": "lmms-lab/DocVQA dataset card",
    "url": "https://huggingface.co/datasets/lmms-lab/DocVQA"
   },
   {
    "accessed": "2026-09-07",
    "title": "inspect_evals: docvqa task README",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/docvqa/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "DocVQA Leaderboard - llm-stats.com",
    "url": "https://llm-stats.com/benchmarks/docvqa"
   }
  ],
  "status": "active",
  "subcategory": "document understanding",
  "summary": "Question answering over scanned and typed document images, scored by fuzzy text match against reference answers.",
  "tags": [
   "ocr",
   "document-understanding",
   "vqa",
   "multimodal"
  ],
  "task_format": "Document image plus a natural-language question in; a short free-text answer out."
 }
}