BBQ (Bias Benchmark for QA)

Multiple-choice QA benchmark testing whether models default to social stereotypes under ambiguous context and can override them once context disambiguates the answer.

Also known as: Bias Benchmark for Question Answering

unassessed

What it measures

BBQ tests whether a language model falls back on social stereotypes when it lacks the information to answer a question, and whether it can set that stereotype aside once the missing fact is supplied. Each item gives the model a short context and a question about two people or groups mentioned in it, with three answer choices: one social group, the other, or "unknown". The nine bias categories are age, disability status, gender identity, nationality, physical appearance, race or ethnicity, religion, socio-economic status, and sexual orientation, plus two intersectional categories combining race with gender and race with socio-economic status. Every item comes in an ambiguous version, which gives no information that would let a careful reader pick a group over "unknown", and a disambiguated version that adds one sentence resolving the question in favour of one of the two groups.

Task format

Three-way multiple-choice QA (two named social groups plus "unknown"), zero- or few-shot, US English

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub