{
 "body": "\n## What it measures\n\nV-FAT asks a multimodal model to name what is actually in an image when language pulls the other way. Level 1 uses atypical images (odd colours, counts, or states) with a neutral question, so the trap is the model's own pretraining prior. Level 2 uses ordinary images plus a prompt that states a false visual premise, isolating sycophancy. Level 3 pairs an atypical image with a misleading prompt that agrees with the prior, so both bias sources fight the pixels at once.\n\nThe set is English visual question answering. Each image\u2013question pair appears as multiple-choice and as a short open-ended answer. The six domains are environment, physical, social, temporal, biological, and functional.\n\n## How it is scored\n\nHeadline scoring is the Visual Robustness Score: the harmonic mean of accuracy against the visual gold answer and resistance to the designed textual trap. A model that is often right only because it followed the language prior is penalised. The paper also reports mean accuracy by format and by level. Multiple-choice items ask for a direct option. Open-ended items ask for a brief phrase, judged automatically by deepseek-chat. Runs used temperature 0 and no extra reasoning unless the model already reasons by default.\n\n## Dataset and licence\n\nThe authors start from about 800 counterfactual images taken from VLind-Bench and WEIRD, filter for clarity, and expand each pair into six tests. The abstract and Section 3.1 report 4,026 instances; the introduction says 4,020. Domain counts in Section 3.1 do not sum to either figure, so treat those counts as published breakdowns rather than a checksum. Half the items are multiple-choice and half open-ended. No dataset licence, download URL, or train split is stated in the paper. The arXiv record uses the site's non-exclusive distribution licence for the article only.\n\n## Who publishes it\n\nZiteng Wang, Yujie He, and Guanliang Li (CUHK Shenzhen), Siqi Yang and Songxiang Liu (Meituan), and Jiaqi Xiong (University of Oxford) released the paper on 8 January 2026. Siqi Yang is listed as corresponding author. No public leaderboard or repository was found from the paper.\n\n## Lineage\n\nV-FAT is a diagnostic overlay on counterfactual visual reasoning, not a general VQA suite. It builds items from VLind-Bench and WEIRD, which do not have pages in this repository. Related hallucination and language-prior tests cited in the paper include HallusionBench, MMStar, WHOOPS!, and IllusionVQA; none of those ids exist here yet. It is not an alias of those evaluations.\n\n## Saturation and contamination\n\nThe paper's 12-model study still separates systems, especially on open-ended answers and on Visual Robustness Score as Level 3 pressure rises. Gemini-3-Flash was unusually stable on Level 3 multiple-choice (95.36%). Seed 1.6 fell to 50.66% on the same slice. Scaling inside Qwen2.5-VL raised accuracy more than VRS. Thinking-mode Qwen3-8B scored worse than the instruct variant. Contamination risk is medium: source images are public, while the three-level prompts are newly generated.\n\n## How to run it\n\nNo lm-evaluation-harness, inspect_evals, HELM, OpenCompass, or BIG-bench task name was found. Reproduce from the paper: temperature 0, both formats, all three levels, deepseek-chat for open-ended grading, and report accuracy and VRS per level. Do not collapse levels into one accuracy number.\n\n## Reading the numbers\n\nA high VRS means the model stayed with the image and refused the trap, not only that it picked a plausible caption. Ordinary VQA accuracy can look strong while VRS is weak. Multiple-choice and open-ended numbers are not interchangeable: option lists raise bias-following errors, while free answers raise unconstrained misses. Compare V-FAT with a standard multimodal accuracy number, and always name the level and format.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "v_fat",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Visual Fidelity Against Text-bias"
  ],
  "category": "multimodal",
  "contamination": {
   "note": "Images come from public counterfactual sets (VLind-Bench and WEIRD). Questions, options, and misleading prompts were generated by MLLMs and checked by a critic model plus expert annotators. Gold answers are implied by the visual fact, so leakage of ordinary VQA labels is less of a concern than reuse of the source images.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": true,
   "size": 4026,
   "size_note": "The abstract and construction section report 4,026 VQA instances; an earlier paragraph says 4,020. Items cover six domains with published counts Environment 882, Physical 354, Social 318, Temporal 195, Biological 186, and Functional 36. Construction starts from about 800 counterfactual images (Figure 6 shows 790 image groups) drawn from VLind-Bench and WEIRD, then expands each pair into six tests (two formats times three bias levels). Multiple-choice and open-ended each make up half of the set.\n",
   "splits": "single reported evaluation set; no train/validation split is described",
   "url": "https://arxiv.org/abs/2601.04897"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-084 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "v_fat",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "V-FAT is a diagnostic visual question-answering test for multimodal large language models. Each item pairs an image with a question that can be answered from pixels, then adds increasing textual pressure: atypical scenes that fight common language priors, misleading instructions that assert a false visual fact, or both at once. The skill is visual fidelity rather than ordinary VQA accuracy, because a linguistically plausible guess can still be wrong relative to the image.\n",
  "metric": {
   "baseline_note": "VRS is the harmonic mean of mean accuracy against the visual gold answer and resistance (one minus the rate of matching the designed textual trap). The paper also reports raw accuracy for multiple-choice and open-ended formats at each of the three levels. Open-ended answers are judged by deepseek-chat. No human baseline is given.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Visual Robustness Score (VRS)",
   "random_baseline": null,
   "unit": "score (0-1)"
  },
  "name": "V-FAT",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2601.04897",
   "title": "V-FAT: Benchmarking Visual Fidelity Against Text-bias",
   "url": "https://arxiv.org/abs/2601.04897",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Ziteng Wang",
    "Yujie He",
    "Guanliang Li",
    "Siqi Yang",
    "Jiaqi Xiong",
    "Songxiang Liu"
   ],
   "org": "The Chinese University of Hong Kong, Shenzhen, with Meituan and the University of Oxford",
   "url": "https://arxiv.org/abs/2601.04897"
  },
  "released": "2026-01",
  "repo_url": "",
  "saturation": {
   "as_of": "2026-01",
   "note": "Gemini-3-Flash reached 95.36% multiple-choice accuracy at Level 3 in the paper, but open-ended scores and Visual Robustness Score remain much lower for most of the 12 models, including large open-weight systems. No independent leaderboard was found.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "V-FAT: Benchmarking Visual Fidelity Against Text-bias (arXiv abs, 2601.04897)",
    "url": "https://arxiv.org/abs/2601.04897"
   },
   {
    "accessed": "2026-09-08",
    "title": "V-FAT HTML full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2601.04897"
   }
  ],
  "status": "active",
  "subcategory": "visual grounding under text bias",
  "summary": "V-FAT tests whether multimodal models answer from the image when corpus priors or misleading prompts conflict with what is shown.",
  "tags": [
   "multimodal",
   "visual-grounding",
   "hallucination",
   "sycophancy",
   "vqa"
  ],
  "task_format": "Image plus a multiple-choice or open-ended question at one of three bias levels; the model must report the visual fact, not the text prior or the prompt's false premise.\n"
 }
}