A BIG-bench task built from the VitaminC dataset that asks a model to judge whether a Wikipedia passage supports, refutes, or gives no information about a claim.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | evidence-based fact verification (claim/evidence entailment) |
| Page status | active |
| Metric | Multiple choice grade (accuracy on the SUPPORTS / REFUTES / NOT ENOUGH INFO 3-way choice) |
| Direction | higher_is_better |
| Unit | accuracy |
| Dataset size | 54668 |
| Dataset licence | CC-BY-SA-3.0 |
| Publisher | Google (BIG-bench collaboration); source dataset by MIT CSAIL (Schuster, Fisch, Barzilay) |
VitaminC Fact Verification presents a claim together with a short piece of evidence text drawn from Wikipedia and asks the model to classify the relationship as "SUPPORTS", "REFUTES", or "NOT ENOUGH INFO". The source VitaminC dataset was built to be contrastive: many claim pairs are nearly identical in wording but differ in the underlying fact, because the evidence comes from real Wikipedia edits that changed a fact, so the task also probes whether a model follows the evidence given rather than relying on facts memorised from pretraining, and whether it can do the numerical or factual comparison needed when the evidence conflicts with what it may already believe.
Zero-shot, 3-way multiple-choice classification, scored with BIG-bench's multiple_choice_grade metric. The task's own convert_to_lm_format.py builds the multiple-choice queries from the underlying VitaminC claim-evidence-label triples.
No model card in ModelSpec reports this benchmark yet.