{
 "body": "\n## What it measures\n\nOCRBench tests whether a large multimodal model can read text embedded in an image and use it to answer\na question, rather than testing general image understanding. It covers five distinct task types: plain\ntext recognition, scene-text-centric visual question answering (reading a sign or label in a photo),\ndocument-oriented visual question answering (forms, receipts, scanned pages), key information extraction\n(pulling a specific field out of a structured document), and handwritten mathematical expression\nrecognition. Combining five task types under one benchmark was a deliberate choice by the authors, who\nfound that models strong on one kind of text-in-image task were often weak on another.\n\n## How it is scored\n\nEach of the 1,000 question-answer pairs is graded on whether the model's answer matches the reference\ntext, and scores are summed across the five task types into an aggregate. A confirmed maximum point total\nfor that aggregate was not found in any source opened during this research, so a bare \"OCRBench score\"\nshould be read against whatever scale the specific reporter states rather than assumed to be a\npercentage. The dataset itself does not define shot count or prompt wording, so those are set by whoever\nruns the evaluation.\n\n## Dataset and licence\n\nThe benchmark combines and re-verifies items from 29 existing OCR-related datasets, including well-known\nscene-text sources such as IIIT5K, SVT, IC13, IC15, SVTP and CT80, plus document and handwriting sources,\ninto 1,000 hand-checked question-answer pairs. The GitHub repository hosting the evaluation code states\nan MIT licence; the redistributed dataset's own terms were not separately confirmed. There is a single\n1,000-item test split, and the questions and reference answers are publicly viewable on Hugging Face.\n\n## Who publishes it\n\nThe paper \"OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models\" is by Yuliang Liu, Zhang\nLi, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin and Xiang\nBai, first posted to arXiv in May 2023 and later published in Science China Information Sciences. The\nsame team maintains the `Yuliang-Liu/MultimodalOCR` repository and a companion results collection on\nHugging Face Spaces.\n\n## Lineage\n\nOCRBench has a direct, larger successor, OCRBench v2 (id: `ocrbench_v2`, no page yet in this repository),\nwhich the authors built with four times as many tasks across 31 scenarios and 10,000 human-verified\nquestion-answer pairs. No predecessor benchmark is recorded, since OCRBench itself is a combination and\nre-verification of 29 prior datasets rather than an extension of one.\n\n## Saturation and contamination\n\nThat the authors built OCRBench v2 at roughly ten times the item count and four times the task count\nsuggests the original 1,000-pair set was starting to run out of room to separate strong models, though no\nsource opened during this research gave a current top score on the original benchmark to confirm that\ndirectly. Contamination risk is mixed and unresolved: several of the 29 source datasets have been public\nfor over a decade, which would normally suggest high risk, but OCRBench's own question-answer curation is\nonly from 2023, and no source read during this research assessed contamination for the combined benchmark\nspecifically.\n\n## How to run it\n\nThe `Yuliang-Liu/MultimodalOCR` repository provides the reference evaluation scripts, and results are\nalso collected on a companion Hugging Face Space per the repository's own documentation. Because the\nbenchmark spans five quite different task types with no fixed prompt template mandated by the dataset,\nand no confirmed maximum score to normalise against, two reporters' aggregate numbers are only safely\ncomparable when both used the same evaluation code and prompt setup.\n\n## Reading the numbers\n\nA high OCRBench score indicates a model can read and reason about text across a wide range of visual\ncontexts, from scene photos to scanned documents to handwriting, rather than excelling at just one OCR\nsub-task. Because the aggregate blends five different task types, two models with the same total can have\nvery different strengths and weaknesses underneath, so it is worth checking a per-task breakdown where\none is available rather than relying on the single number. The existence of the larger OCRBench v2 is\nalso a signal that the original benchmark's headroom for separating top models may be narrowing.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ocrbench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL 8B Instruct",
   "model_id": "qwen/qwen3-vl-8b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 89.6,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4",
   "model_id": "openbmb/minicpm-v-4",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 89.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5",
   "model_id": "openbmb/minicpm-v-4-5",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 89.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "MiniCPM V 4 5 gguf",
   "model_id": "openbmb/minicpm-v-4-5-gguf",
   "provider": "openbmb",
   "provider_display": "OpenBMB",
   "score": 89.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 88.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 VL 32B Instruct",
   "model_id": "qwen/qwen3-vl-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 87.5,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 7B Instruct",
   "model_id": "qwen/qwen2-5-vl-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 86.4,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "NVLM D 72B",
   "model_id": "nvidia/nvlm-d-72b",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 85.2,
   "source": "multimodal-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 78.8,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.5,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 73.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 73.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 73.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 73.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 68.9,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.6,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  }
 ],
 "page": {
  "aliases": [],
  "category": "multimodal",
  "contamination": {
   "note": "Several of the 29 source datasets (for example IIIT5K, SVT, IC13, IC15) have been public for over a decade, but OCRBench's specific 1,000-item question-answer curation dates only to 2023; no source opened during research assessed contamination for the resulting benchmark specifically.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 1000,
   "size_note": "1,000 manually verified question-answer pairs, combined from 29 existing OCR-related datasets.",
   "splits": "Single test split of 1,000 items.",
   "url": "https://huggingface.co/datasets/echo840/OCRBench"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice H",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Reference evaluation code and a companion leaderboard are maintained in the Yuliang-Liu/MultimodalOCR GitHub repository, with results also collected on a Hugging Face Space per the repository's own README."
  },
  "id": "ocrbench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "ocrbench_v2"
   ],
   "variants": []
  },
  "measures": "OCRBench evaluates optical character recognition ability in large multimodal models across five task types: plain text recognition, scene-text-centric visual question answering, document-oriented visual question answering, key information extraction, and handwritten mathematical expression recognition. Items are drawn from 29 existing OCR-related datasets and re-verified by hand into 1,000 question-answer pairs, so the benchmark is broader than any single source dataset while staying small enough to evaluate cheaply.\n",
  "metric": {
   "baseline_note": "The paper and repository report a summed score across the five task types but a confirmed maximum point total was not found in any source opened during research; comparisons should use a reporter's own stated scale rather than assuming one.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "aggregate score across five task types",
   "random_baseline": null,
   "unit": "points"
  },
  "name": "OCRBench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2305.07895",
   "title": "OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models",
   "url": "https://arxiv.org/abs/2305.07895",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Yuliang Liu",
    "Zhang Li",
    "Mingxin Huang",
    "Biao Yang",
    "Wenwen Yu",
    "Chunyuan Li",
    "Xucheng Yin",
    "Cheng-lin Liu",
    "Lianwen Jin",
    "Xiang Bai"
   ],
   "org": "",
   "url": "https://github.com/Yuliang-Liu/MultimodalOCR"
  },
  "released": "2023-05",
  "repo_url": "https://github.com/Yuliang-Liu/MultimodalOCR",
  "saturation": {
   "as_of": "",
   "note": "A successor, OCRBench v2, was built with four times as many tasks (31 scenarios, 10,000 human-verified pairs) specifically to be more comprehensive than the original 1,000-pair set, which suggests the original was becoming too easy or too narrow for distinguishing top models by the time the successor was made. No current top score on the original benchmark was confirmed from a source opened during this research.",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models",
    "url": "https://arxiv.org/abs/2305.07895"
   },
   {
    "accessed": "2026-09-07",
    "title": "GitHub - Yuliang-Liu/MultimodalOCR",
    "url": "https://github.com/Yuliang-Liu/MultimodalOCR"
   },
   {
    "accessed": "2026-09-07",
    "title": "MultimodalOCR/OCRBench/README.md at main",
    "url": "https://github.com/Yuliang-Liu/MultimodalOCR/blob/main/OCRBench/README.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "echo840/OCRBench \u00b7 Datasets at Hugging Face",
    "url": "https://huggingface.co/datasets/echo840/OCRBench"
   }
  ],
  "status": "active",
  "subcategory": "OCR and document understanding",
  "summary": "1,000 hand-verified questions across five OCR task types, testing whether a multimodal model can read and reason about text embedded in images.",
  "tags": [
   "ocr",
   "multimodal",
   "text-recognition",
   "document-understanding",
   "handwriting"
  ],
  "task_format": "Open-ended, short-answer visual question answering over an image containing text; the model must read the text to answer."
 }
}