A HELM scenario that gives a model a generated statement and its cited source and asks it to judge whether the source fully, partially, or does not support the statement.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | citation verification (statement-to-source support classification) |
| Page status | active |
| Metric | Exact match / quasi-exact match against the human-annotated 3-way label |
| Direction | higher_is_better |
| Unit | accuracy |
| Dataset size | 11037 |
| Dataset licence | MIT |
| Publisher | Stanford NLP Group (source dataset); Stanford CRFM (HELM implementation) |
Verifiability Judgment presents a statement, taken from a real generative search engine's response, together with the source document it cited, and asks the model to classify the citation relationship as "fully supports", "partially supports", or "does not support" the statement. It tests whether a model can perform the citation-verification step that underlies retrieval-augmented and search-grounded systems: reading a passage and judging whether it actually substantiates a specific claim, rather than merely appearing topically related.
Zero-shot (or few-shot, per HELM's default adapter configuration) text generation: HELM prompts the model with the statement and source under instructions to judge support as "fully supports", "partially supports" or "does not support", generating up to 10 tokens, and scores the output against the human-annotated label with exact match and quasi-exact match.
No model card in ModelSpec reports this benchmark yet.