EsBBQ (Spanish Bias Benchmark for Question Answering)

Spanish multiple-choice QA, parallel with CaBBQ, testing stereotype use under ambiguous context and accuracy once the context names the answer.

Also known as: Spanish Bias Benchmark for Question Answering, Spanish BBQ, EsBBQ

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	safety
Subcategory	social bias in Spanish question answering
Page status	active
Metric	acc_ambig and acc_disambig (higher better); bias_score_ambig and bias_score_disambig (near 0 better)
Direction	higher_is_better
Unit	%
Dataset size	27320
Dataset licence	CC-BY-4.0
Publisher	Barcelona Supercomputing Center, Language Technologies Unit

What it measures

EsBBQ asks whether a model falls back on stereotypes about groups in Spain when a Spanish context is under-informative, and whether it can ignore that stereotype once a disambiguating sentence is added. Each item has a context, a question, and three answers (two groups plus an unknown option). Ten categories cover age, disability, gender, LGBTQIA, nationality, appearance, race/ethnicity, religion, SES, and Spanish region. It is a cultural adaptation of English BBQ, not a translation-only copy, and it is fully parallel with Catalan CaBBQ.

Task format

Three-way multiple-choice QA in Spanish. lm-eval group `esbbq` scores by log-likelihood over ans0, ans1, and several unknown phrasings collapsed to index 2. Prompts use Contexto / Pregunta / Respuesta.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub