{
 "body": "\n## What it measures\n\nVerifiability Judgment gives a model a statement, originally produced by a real generative search\nengine, together with the source document that engine cited for it, and asks the model to classify\nwhether the source \"fully supports\", \"partially supports\", or \"does not support\" the statement.\nThe task isolates the citation-verification step underlying retrieval-augmented generation and\nsearch-grounded answering: it tests reading comprehension for entailment between a specific claim\nand a candidate source passage, not whether the model can generate a well-cited answer itself.\n\n## How it is scored\n\nHELM prompts the model with the statement and source text under an instruction to answer with one\nof the three support levels, generating up to 10 tokens, and scores the output against the\nhuman-annotated gold label using exact match and quasi-exact match. The three label categories are\nunevenly distributed in the source data (complete support makes up roughly 70 to 75% of examples\nin each split), so a naive baseline that always predicts \"fully supports\" would score well above\nthe 1-in-3 chance level a uniform guess would imply. No human accuracy baseline for this\nclassification task was found in the sources read for this page.\n\n## Dataset and licence\n\nThe underlying dataset holds 11,037 human-annotated statement-source pairs split 8,834 train,\n1,106 dev, and 1,097 test, released under an MIT licence in Nelson Liu's GitHub repository\naccompanying the paper. Each JSONL record includes the original user query, the generative search\nengine's statement, the cited source's metadata (title, author, date, URL) and text, the\ncrowdworker-assigned support label, and annotator-identified evidence excerpts. HELM downloads and\nuses these same splits directly rather than a resampled subset.\n\n## Who publishes it\n\nThe source dataset and study come from Nelson F. Liu, Tianyi Zhang and Percy Liang of Stanford,\npublished as \"Evaluating Verifiability in Generative Search Engines\" at Findings of EMNLP 2023. The\npaper's main contribution was auditing four commercial generative search engines (Bing Chat,\nNeevaAI, Perplexity.ai and YouChat) for citation quality; the human judgments collected for that\naudit were repurposed by Stanford CRFM as a standalone classification scenario in the HELM\nbenchmark suite, which is what this page documents.\n\n## Lineage\n\nVerifiability Judgment has no predecessor or successor benchmark tracked in this repository. It is\na HELM-specific repackaging of human annotations originally collected to audit real generative\nsearch engines rather than to train or evaluate general-purpose fact-verification models, so its\nlabel distribution and source-passage style reflect real search-engine outputs from 2023 rather\nthan a benchmark purpose-built for balanced classification.\n\n## Saturation and contamination\n\nNo maintained leaderboard or per-model score table specific to this HELM scenario was found in the\nsources read for this page, so whether current models have saturated it is not established. The\nsource paper's own finding, that the audited search engines fully supported only about half their\nsentences with citations, describes those systems' citation behavior rather than a model's accuracy\nat this classification task, so it should not be read as a saturation signal for this benchmark.\nContamination risk is medium: the complete dataset with gold labels, including the test split HELM\nuses, has been openly available on GitHub since 2023.\n\n## How to run it\n\nRun via HELM's `verifiability_judgment` run spec, which uses `VerifiabilityJudgementScenario` to\nbuild prompts from the source repository's train/dev/test files and a generation adapter that asks\nfor one of the three support levels in up to 10 tokens, scored with exact match and quasi-exact\nmatch. There is no equivalent task in lm-evaluation-harness, inspect_evals, OpenCompass or\nBIG-bench as of the source read for this page; comparing a score to any other citation-verification\nbenchmark requires checking that the label set and prompting instructions match.\n\n## Reading the numbers\n\nA high score means a model reliably distinguishes whether a source passage actually backs up a\nspecific claim, a useful proxy for how well a model could self-check citations in a\nretrieval-augmented pipeline. Because the label distribution skews heavily toward \"complete\nsupport,\" a raw accuracy number can be inflated by a model that defaults to that answer; comparing\nper-class performance or checking against the majority-class baseline gives a more honest read.\nThe benchmark does not test whether a model can generate accurate citations itself, only whether it\ncan judge citations that already exist.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "verifiability_judgment",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "The full dataset, with gold labels for all three splits, has been publicly available under an MIT licence on GitHub since 2023, including the test split HELM evaluates on, so exact statement-source-label triples could appear in training data for models trained on broad web or GitHub scrapes.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 11037,
   "size_note": "11,037 human-annotated statement-source pairs total across an 80/10/10 split: 8,834 train, 1,106 dev, 1,097 test, per the source GitHub repository's data files. Label counts: train 6,415 complete / 1,552 partial / 867 no support; dev 830 / 165 / 111; test 797 / 183 / 117. HELM's scenario downloads and uses these same train/dev/test splits.\n",
   "splits": "train (8,834), dev (1,106), test (1,097), as released by the source repository",
   "url": "https://github.com/nelson-liu/evaluating-verifiability-in-generative-search-engines"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-008 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-008"
  },
  "harness": {
   "bigbench": "",
   "helm": "verifiability_judgment",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "verifiability_judgment",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Verifiability Judgment presents a statement, taken from a real generative search engine's response, together with the source document it cited, and asks the model to classify the citation relationship as \"fully supports\", \"partially supports\", or \"does not support\" the statement. It tests whether a model can perform the citation-verification step that underlies retrieval-augmented and search-grounded systems: reading a passage and judging whether it actually substantiates a specific claim, rather than merely appearing topically related.\n",
  "metric": {
   "baseline_note": "Three roughly-defined label categories give a naive uniform-guess baseline near 1/3, though the label distribution is imbalanced toward \"complete support\" (about 73% of the source dataset's examples), so a model that always answers \"fully supports\" would score close to that proportion rather than 0.333. No human baseline accuracy figure was found in the sources read for this page; the source dataset's labels are themselves derived from human crowdworker annotation, which HELM treats as ground truth.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Exact match / quasi-exact match against the human-annotated 3-way label",
   "random_baseline": 0.333,
   "unit": "accuracy"
  },
  "name": "Verifiability Judgment",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2304.09848",
   "title": "Evaluating Verifiability in Generative Search Engines",
   "url": "https://arxiv.org/abs/2304.09848",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Nelson F. Liu",
    "Tianyi Zhang",
    "Percy Liang"
   ],
   "org": "Stanford NLP Group (source dataset); Stanford CRFM (HELM implementation)",
   "url": "https://github.com/nelson-liu/evaluating-verifiability-in-generative-search-engines"
  },
  "released": "2023-04",
  "repo_url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/verifiability_judgment_scenario.py",
  "saturation": {
   "as_of": "",
   "note": "No maintained leaderboard or published per-model score table specifically for this HELM scenario was found in the sources read for this page, so saturation status is not established. The source paper's own headline finding, that four commercial generative search engines fully supported only 51.5% of their sentences on average, describes citation quality of those systems' outputs, not model performance at the classification task this benchmark measures.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM VerifiabilityJudgementScenario source: label mapping (complete_support/partial_support/no_support), prompt construction, train/dev/test download from source GitHub repo",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/verifiability_judgment_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM classic_run_specs.py get_verifiability_judgment_spec(): generation adapter, instructions text, max_tokens=10, exact_match and quasi_exact_match metrics",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/run_specs/classic_run_specs.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Evaluating Verifiability in Generative Search Engines (Liu, Zhang, Liang, 2023, Findings of EMNLP 2023): defines verifiability via citation recall/precision, evaluates Bing Chat, NeevaAI, Perplexity.ai, YouChat; average 51.5% of sentences fully supported by citations",
    "url": "https://arxiv.org/abs/2304.09848"
   },
   {
    "accessed": "2026-09-08",
    "title": "Source repository README and data files: MIT licence, 80/10/10 train/dev/test split (8,834/1,106/1,097), per-split label counts, JSONL data format with query, statement, source text, and judgment fields",
    "url": "https://github.com/nelson-liu/evaluating-verifiability-in-generative-search-engines"
   }
  ],
  "status": "active",
  "subcategory": "citation verification (statement-to-source support classification)",
  "summary": "A HELM scenario that gives a model a generated statement and its cited source and asks it to judge whether the source fully, partially, or does not support the statement.",
  "tags": [
   "fact-verification",
   "citation",
   "retrieval-augmented-generation",
   "helm"
  ],
  "task_format": "Zero-shot (or few-shot, per HELM's default adapter configuration) text generation: HELM prompts the model with the statement and source under instructions to judge support as \"fully supports\", \"partially supports\" or \"does not support\", generating up to 10 tokens, and scores the output against the human-annotated label with exact match and quasi-exact match.\n"
 }
}