{
 "body": "\n## What it measures\n\nFACTS Grounding gives the model a long document and a user request, then asks for a long answer that uses only that document. A single ungrounded claim fails the item.\n\nTasks include summarising, extracting facts and rewriting, across finance, technology, retail, medical and legal pages. Annotators dropped creative writing, expert-only items, maths and OCR-broken PDFs. The skill is faithfulness to provided context, not being right about the world.\n\n## How it is scored\n\nv1 uses three judges (Gemini 1.5 Pro, GPT-4o, Claude 3.5 Sonnet). Each judge first decides whether the answer actually addresses the request. If all three call a major instruction miss, the answer is ineligible and counts as inaccurate. Surviving answers are labelled accurate only if every information-bearing claim is grounded.\n\nThe published factuality score is the mean of the three judges after that filter. The paper also reports an unadjusted score and a Condorcet fused rank. Self-preference was measured at about +3.23% when a model judged itself. Eligibility filtering cut scores by roughly 1-5 points and swapped Gemini 1.5 Flash from fused rank 1 to rank 2.\n\n## Dataset and licence\n\n1,719 examples: 860 public, 859 private. Hugging Face `google/FACTS-grounding-public` is tagged `license: cc-by-4.0` and shows 860 public rows. The dataset card also ships judge prompt templates. The paper licence is CC BY 4.0.\n\nThere are no gold long-form answers in the public CSV. Scoring is always judge-based. The private split is how Kaggle scores unofficial submissions.\n\n## Who publishes it\n\nGoogle DeepMind, Google Research, Google Cloud and Kaggle. Equal-contribution leads on the paper include Jacovi, Wang, Alberti, Tao, Lipovetz, Olszewska, Haas, Liu, Keating and Das. Hugging Face created the public dataset on 2024-12-18. arXiv v1 is 6 January 2025.\n\nKaggle hosts the leaderboard and starter notebook. Live rows may move after the paper table.\n\n## Lineage\n\nThis page is Grounding v1 (arXiv:2501.03200). [the_facts_leaderboard](the_facts_leaderboard.md) is the December 2025 FACTS suite. That suite keeps these prompts as Grounding v2 but swaps judges to Gemini 2.5 Flash and GPT-5.\n\nRelated but different: [simpleqa](simpleqa.md) is closed-book short factoids. Long-form web factuality papers (FActScore, LongFact) are not this document-grounding task.\n\nThe Hugging Face citation block titles the dataset \"FACTS Leaderboard\" even though this file is the grounding-only v1 eval. Use the arXiv title when you need to disambiguate.\n\n## Saturation and contamination\n\nPaper-era top score is 83.6, with a 22-point spread down to o1-preview. That is not saturated. Grounding v2 later reports lower absolute scores with newer judges; do not mix v1 83.6 with suite Grounding columns.\n\nDocuments may be in pretraining. The authors argue the novel requests still test grounding, and that using memorised facts against the document should hurt. Private items reduce public overfitting.\n\n## How to run it\n\nPublic prompts: Hugging Face CSV. Starter code: the Kaggle notebook named in `repo_url`. There is no confirmed lm-eval or inspect_evals task id.\n\nA local run that changes judge models, drops the eligibility filter, or scores only the public split will not match Kaggle. v1 numbers need the three v1 judges; v2 numbers need the suite judges.\n\n## Reading the numbers\n\nA high v1 score means judges found the long answer both on-task and fully supported by the supplied document. It does not mean the model is generally truthful without a document, and it does not mean it can search.\n\nShort, empty answers can look grounded; that is why eligibility exists. When a vendor quotes \"FACTS\", check whether they mean this grounding leaderboard or the four-track [the_facts_leaderboard](the_facts_leaderboard.md) average.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "the_facts_grounding_leaderboard",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "FACTS Grounding",
   "FACTS Grounding v1"
  ],
  "category": "generation",
  "contamination": {
   "note": "Context documents were scraped from the web and may appear in pretraining. User requests and the \"context-only\" system instructions are new. A private split exists. The metric penalises using extra-document knowledge even when that knowledge is true.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 1719,
   "size_note": "860 public Open examples plus 859 private Blind examples. Hugging Face google/FACTS-grounding-public hosts the 860-row public CSV (system_instruction, user_request, context_document, full_prompt) and evaluation_prompts.csv.\n",
   "splits": "Open (public, n=860) / Blind (private, n=859)",
   "url": "https://huggingface.co/datasets/google/FACTS-grounding-public"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-082 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Kaggle starter notebook (facts-grounding-benchmark-starter-code)"
  },
  "id": "the_facts_grounding_leaderboard",
  "last_updated": "2025-01",
  "leaderboard_url": "https://www.kaggle.com/facts-leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "the_facts_leaderboard"
   ],
   "variants": []
  },
  "measures": "FACTS Grounding checks scenario-one factuality: every claim in a long-form answer must be supported by a document in the prompt, not by parametric world knowledge. Each example has a system instruction (\"use only this context\"), a user request (QA, summary, rewrite) and a web-sourced document up to 32k tokens (mean about 2.5k). The request must be non-trivial and must not need extra-document expertise, maths, or creative writing. This is attribution to provided context, not closed-book factoids and not web search.\n",
  "metric": {
   "baseline_note": "v1 judges are Gemini 1.5 Pro, GPT-4o and Claude 3.5 Sonnet, each with a prompt picked by Macro-F1 on a 402-example private set. Unadjusted score is the mean of the three judges' accurate rates. A response is ineligible only if all three judges mark a major instruction-following failure; ineligible replies count as inaccurate. Paper Table 6 top row: Gemini 2.0 Flash Experimental 83.6 after that filter.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "final factuality score (share of responses that are eligible and fully grounded, averaged over three judges)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "The FACTS Grounding Leaderboard",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2501.03200",
   "title": "The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input",
   "url": "https://arxiv.org/abs/2501.03200",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Alon Jacovi",
    "Andrew Wang",
    "Chris Alberti",
    "Connie Tao",
    "Jon Lipovetz",
    "Kate Olszewska",
    "Lukas Haas",
    "Michelle Liu",
    "Nate Keating",
    "Dipanjan Das"
   ],
   "org": "Google DeepMind, Google Research, Google Cloud, and Kaggle",
   "url": "https://www.kaggle.com/facts-leaderboard"
  },
  "released": "2024-12",
  "repo_url": "https://www.kaggle.com/code/andrewmingwang/facts-grounding-benchmark-starter-code",
  "saturation": {
   "as_of": "2025-01",
   "note": "Paper Table 6 (final score after eligibility filter) lists Gemini 2.0 Flash Experimental at 83.6 and o1-preview at 61.7. Grounding v2 in the later FACTS suite uses different judges and is not this number.\n",
   "status": "open",
   "top_score": 83.6
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "FACTS Grounding paper (arXiv:2501.03200v1)",
    "url": "https://arxiv.org/abs/2501.03200"
   },
   {
    "accessed": "2026-09-08",
    "title": "FACTS Grounding HTML full text",
    "url": "https://ar5iv.labs.arxiv.org/html/2501.03200"
   },
   {
    "accessed": "2026-09-08",
    "title": "google/FACTS-grounding-public dataset card",
    "url": "https://huggingface.co/datasets/google/FACTS-grounding-public"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API metadata (license cc-by-4.0, 860 public rows)",
    "url": "https://huggingface.co/api/datasets/google/FACTS-grounding-public"
   },
   {
    "accessed": "2026-09-08",
    "title": "FACTS Grounding Kaggle leaderboard",
    "url": "https://www.kaggle.com/facts-leaderboard"
   }
  ],
  "status": "active",
  "subcategory": "long-form answers grounded in a supplied document (up to 32k tokens)",
  "summary": "Google FACTS Grounding scores whether long-form answers stay faithful to a supplied document, using an ensemble of LLM judges plus an eligibility filter.",
  "tags": [
   "factuality",
   "grounding",
   "long-context",
   "llm-as-judge"
  ],
  "task_format": "Single-turn generation: system instruction plus user request plus full document. Judges then run two stages, eligibility then span/response-level grounding.\n"
 }
}