{
 "body": "## What it measures\n\nGenAI-Bench evaluates text-to-visual generation on compositional prompts. Prompts require models to place attributes, relations, and higher-order concepts such as logic and comparison into an image or video. The benchmark therefore tests whether a model follows the meaning of a prompt, not only whether it produces a realistic picture.\n\nThe project also evaluates automated metrics against human ratings. GenAI-Rank extends the work to ranking images generated from the same prompt.\n\n## How it is scored\n\nThe primary evidence is human alignment ratings. The paper compares VQAScore, which measures whether a VQA model sees an image as depicting the prompt, with CLIPScore, PickScore, HPSv2, and ImageReward. It reports that VQAScore improves ranking and can select among three to nine candidate images. Exact rating scales and video-specific aggregation are not established in the abstract.\n\n## Dataset and licence\n\nThe paper says GenAI-Rank contains over 40,000 human ratings and that over 80,000 human ratings will be released. The project page and code are public. The consulted sources do not establish one consolidated licence for prompts, generated media, and ratings, so licensing should be checked in the release.\n\n## Who publishes it\n\nBaiqi Li and ten coauthors introduced GenAI-Bench in a 2024 arXiv paper. The authors link the official project page and open-source dataset, model, and code. No single current leaderboard is identified.\n\n## Lineage\n\nGenAI-Bench extends an earlier introduction cited by the paper. GenAI-Rank is its ranking-focused successor or variant, using human ratings to evaluate metrics on same-prompt image comparisons. The two should not be conflated.\n\n## Saturation and contamination\n\nThe paper finds substantial differences among automated metrics and reports that VQAScore better tracks human judgments on compositional prompts. This indicates an open measurement problem. Public release of prompts and ratings creates medium contamination risk for future generators.\n\n## How to run it\n\nUse the official prompts, generation settings, candidate-selection protocol, and human-rating rubric. Report image versus video, prompt subset, number of candidates, and whether VQAScore or human ratings are used. Metric comparisons require identical generated candidates.\n\n## Reading the numbers\n\nA high human alignment score means the output follows the tested prompt to human raters. It does not establish broad visual quality, originality, or temporal coherence. Automated metric scores are proxies whose agreement can vary by prompt type. Inspect attribute and relation failures alongside aggregate ratings.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "genai_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "multimodal",
  "contamination": {
   "note": "Human ratings and prompts are released or planned for release, so later training exposure is possible.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "modalities": [
    "text",
    "image",
    "video"
   ],
   "public_test_set": true,
   "size": 40000,
   "size_note": "GenAI-Rank contains over 40,000 human ratings; the paper also says over 80,000 ratings will be released.",
   "url": "https://linzhiqiu.github.io/"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-stream-a-004 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "other": "Official GenAI-Bench and GenAI-Rank code and rating protocol."
  },
  "id": "genai_bench",
  "lineage": {
   "successors": [
    "genai_rank"
   ]
  },
  "measures": "GenAI-Bench tests whether generated visuals follow compositional prompts involving attributes, relationships, logic, and comparison. It evaluates image and video generation and studies whether VQAScore agrees with human judgments.",
  "metric": {
   "baseline_note": "The paper compares VQAScore with other automated metrics; no universal maximum is established.",
   "direction": "higher_is_better",
   "name": "human alignment rating",
   "unit": "rating"
  },
  "name": "GenAI-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2406.13743",
   "title": "GenAI-Bench: Evaluating and Improving Compositional Text-to-Visual Generation",
   "url": "https://arxiv.org/abs/2406.13743",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Baiqi Li",
    "Zhiqiu Lin",
    "Deepak Pathak",
    "Jiayao Li",
    "Yixin Fei",
    "Kewen Wu",
    "Tiffany Ling",
    "Xide Xia",
    "Pengchuan Zhang",
    "Graham Neubig",
    "Deva Ramanan"
   ],
   "org": "GenAI-Bench authors",
   "url": "https://arxiv.org/abs/2406.13743"
  },
  "released": "2024-06",
  "repo_url": "https://linzhiqiu.github.io/",
  "saturation": {
   "note": "The paper reports meaningful gaps among metrics and generation systems.",
   "status": "open"
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "GenAI-Bench paper",
    "url": "https://arxiv.org/abs/2406.13743"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official GenAI-Bench project page",
    "url": "https://linzhiqiu.github.io/"
   }
  ],
  "summary": "GenAI-Bench evaluates compositional text-to-image and text-to-video generation with human ratings and automated metric comparisons.",
  "tags": [
   "text-to-image",
   "text-to-video",
   "compositionality",
   "human-evaluation"
  ],
  "task_format": "Text prompts paired with generated images or videos and human preference ratings."
 }
}