Misconceptions

A 219-item BIG-bench true/false task that asks whether a short English statement is a popular misconception or a fact.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	knowledge
Subcategory	BIG-bench true/false popular misconceptions (219 items)
Page status	unknown
Metric	multiple_choice_grade
Direction	higher_is_better
Unit	%
Dataset size	219
Dataset licence	Apache-2.0
Publisher	Google (BIG-bench collaboration)

What it measures

misconceptions presents one English sentence and asks the model to output T or F. The items are rewritten from lists of common errors, not copied trivia questions. The authors (including Hendrycks, Mazeika and Zou) frame it as a truthfulness probe: larger web-trained models may mimic the popular falsehood. It is not [TruthfulQA](truthfulqa.md), which uses 817 trap questions and a different scoring stack.

Task format

Two-option multiple choice (T/F), preferred metric `multiple_choice_grade`. task_prefix: "Is the following statement true or false? Output T for true and F for false." Canary GUID embedded. JSON task; 219 dummy-model queries in the README header.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub