{
 "body": "## What it measures\n\nMRAG-Bench evaluates retrieval-augmented vision-language models. It focuses on cases where retrieved images are more useful or easier to access than textual knowledge, including multiple views of a subject. A system retrieves visual evidence and answers a human-annotated multiple-choice question.\n\nThe benchmark is vision-centric: it tests both retrieval usefulness and the model\u2019s ability to incorporate visual evidence into its answer.\n\n## How it is scored\n\nThe main measure is multiple-choice accuracy under the benchmark\u2019s retrieval conditions. The paper compares text augmentation, image augmentation, and ground-truth information. It reports 10 open-source and four proprietary LVLMs. GPT-4o improved 5.82% with ground-truth visual information, compared with 33.16% for human participants in the reported analysis.\n\n## Dataset and licence\n\nMRAG-Bench contains 16,130 images and 1,353 human-annotated questions across nine scenarios. The public Hugging Face card is the dataset source. The consulted sources do not establish the complete licence text or whether every answer split is public, so those details should be checked before redistribution.\n\n## Who publishes it\n\nWenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun Peng introduced the benchmark in a paper accepted to ICLR 2025. The dataset is associated with the uclanlp Hugging Face organisation. No independent leaderboard is established.\n\n## Lineage\n\nMRAG-Bench is a standalone multimodal retrieval benchmark. It extends text-centric retrieval-augmented QA by testing visual evidence. The paper does not identify a successor benchmark.\n\n## Saturation and contamination\n\nThe reported gap between models and humans when using ground-truth visual information shows that visual retrieval use remains open. Because the dataset is public, training exposure is possible, but the paper does not prove contamination for evaluated models. Risk is medium.\n\n## How to run it\n\nUse the official images, questions, retrieval setup, and multiple-choice scorer. Report whether evidence is retrieved or ground truth, the number of images shown, retrieval model, prompt, and LVLM. Keep scenario-level results because aggregate accuracy can conceal retrieval failure.\n\n## Reading the numbers\n\nA high accuracy indicates successful use of the tested visual evidence and answer choices. It does not prove robust multimodal retrieval in unseen domains. Compare text and image augmentation, and separate retrieval quality from answer reasoning. The human comparison is a useful ceiling signal but is not a model baseline for every protocol.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mrag_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "multimodal",
  "contamination": {
   "note": "The dataset is public; the paper does not establish model-specific training exposure.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 1353,
   "size_note": "1,353 human-annotated questions and 16,130 images across 9 scenarios.",
   "url": "https://huggingface.co/datasets/uclanlp/MRAG-Bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-stream-a-004 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "other": "Official MRAG-Bench dataset and retrieval/evaluation code."
  },
  "id": "mrag_bench",
  "measures": "MRAG-Bench tests retrieval-augmented multimodal models on scenarios where images provide more useful evidence than text. It contains 16,130 images and 1,353 human-annotated multiple-choice questions across nine scenarios.",
  "metric": {
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "multiple-choice accuracy",
   "unit": "percent"
  },
  "name": "MRAG-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2410.08182",
   "title": "MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models",
   "url": "https://arxiv.org/abs/2410.08182",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Wenbo Hu",
    "Jia-Chen Gu",
    "Zi-Yi Dou",
    "Mohsen Fayyaz",
    "Pan Lu",
    "Kai-Wei Chang",
    "Nanyun Peng"
   ],
   "org": "MRAG-Bench authors",
   "url": "https://arxiv.org/abs/2410.08182"
  },
  "released": "2024-10",
  "saturation": {
   "note": "The paper reports that leading LVLMs still struggle to exploit retrieved visual knowledge.",
   "status": "open"
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MRAG-Bench paper",
    "url": "https://arxiv.org/abs/2410.08182"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official MRAG-Bench dataset card",
    "url": "https://huggingface.co/datasets/uclanlp/MRAG-Bench"
   }
  ],
  "subcategory": "retrieval-augmented generation",
  "summary": "MRAG-Bench evaluates whether vision-language models can retrieve and use visual knowledge for multimodal question answering.",
  "tags": [
   "multimodal",
   "retrieval",
   "visual-question-answering"
  ],
  "task_format": "Image retrieval plus multiple-choice visual question answering."
 }
}