Fact-Checking (BIG-bench)

A BIG-bench true/false fact-checking task over Wikipedia FEVER claims, COVID-19 scientific claims, and Politifact items.

Also known as: BIG-bench fact_checker, Fact-Checking

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
SubcategoryBIG-bench true/false fact-checking (FEVER, COVID-19 scientific, Politifact)
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size7154
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration); HKUST authors

What it measures

fact_checker asks whether a short English claim is true or false, without retrieved evidence. Three subtasks share that format: FEVER Wikipedia claims (supports/refutes mapped to true/false; "not enough info" dropped), COVID-19 scientific myths, and Politifact political claims balanced across parties. The skill is parametric fact-checking, not evidence-based FEVER verification with Wikipedia pages. It is not [bbh](bbh.md).

Task format

Two-option multiple choice (true/false), preferred metric multiple_choice_grade. Every counted JSON item uses a dated prefix: "On <date>, the following claim was made: … Q: Was this claim true or false?" append_choices_to_input is false. Canary GUID embedded. Keywords include zero-shot, one-shot, and many-shot. The README still shows an older undated "Claim: … Is this claim true or false?" prompt that the merged JSON no longer uses.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub