{
 "body": "\n## What it measures\n\nUnQover tests whether a QA model assigns stereotype-linked scores when a paragraph leaves the answer unresolved. A typical item names two people with different protected properties and asks about an attribute. Both candidate spans are plausible, so a fair model should not prefer one based only on the protected property.\n\nIt covers gender-occupation, nationality, ethnicity and religion. Candidate order and question polarity are perturbed to separate bias from positional dependence and failure to understand the queried attribute.\n\n## How it is scored\n\nThe task reports bias intensity, count-based bias, average answer probability, positional error and attributive error. Headline metrics are fairness, one minus bias intensity, and consistency, which combines answer probability with the two error measures. Fairness and consistency range from 0 to 1 and are higher-is-better; components use their own directions.\n\nCandidate probabilities need not sum to one. High fairness can be misleading when both candidates receive low probability or the model is inconsistent, so report consistency and answer probability too.\n\n## Dataset and licence\n\nBIG-bench generates examples from templates for four stereotype classes. Sources include occupations, manually selected attributes, names, nationalities, ethnicities and religions. The generated header reports 10,552,928 multiple-choice dummy-model queries, not a conventional train/test split.\n\nThe opened sources do not establish one licence for the combined generated data. Source materials may have separate terms.\n\n## Who publishes it\n\nBIG-bench credits Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal and Vivek Srikumar. The underlying paper appeared in Findings of EMNLP 2020. The AllenAI repository contains implementation and metrics. No current standalone leaderboard was established.\n\n## Lineage\n\nUnQover is based on the 2020 paper and appears as a BIG-bench task. Internally, task.py scores four subtasks keyed gender, nationality, ethnicity and religion, but none of these has its own page or a separate BIG-bench id in this repository. UnQover is related to BBQ and StereoSet but uses underspecified span QA and its own fairness and consistency metrics. No successor was established.\n\n## Saturation and contamination\n\nTemplates and source lists are public, so contamination risk is high. The paper discusses robustness to a few memorized examples but not full-task training. Current saturation is unknown.\n\n## How to run it\n\nRun BIG-bench task unqover with span scoring and candidate-order and polarity perturbations. Record model type, span scoring method and aggregation over the four datasets. Do not replace fairness with ordinary QA accuracy.\n\n## Reading the numbers\n\nHigher fairness means lower measured stereotype-linked bias on this template set. It does not show that a model is free of social bias. Inspect consistency and answer probability before interpreting fairness. Pair UnQover with BBQ and evaluations across cultures.\n\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "unqover",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "UNQOVERing Stereotyping Biases via Underspecified Questions"
  ],
  "category": "safety",
  "contamination": {
   "note": "Templates and source lists are public; the paper does not establish resistance to full-task training.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 10552928,
   "size_note": "BIG-bench's generated header reports 10,552,928 multiple-choice dummy-model queries.",
   "splits": "generated gender-occupation, nationality, ethnicity and religion datasets; no train/test split stated",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/unqover"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-new-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-003"
  },
  "harness": {
   "bigbench": "unqover",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "unqover",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "big_bench",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "A paragraph names two candidates and asks an underspecified question. The model scores both spans even though context does not justify choosing either. Perturbations isolate stereotype bias from positional and question-attribute errors.",
  "metric": {
   "baseline_note": "The README defines fairness and consistency but gives no random or human baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "fairness (1 minus bias intensity); consistency alongside it",
   "random_baseline": null,
   "unit": "score"
  },
  "name": "UnQover",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "UNQOVERing Stereotyping Biases via Underspecified Questions",
   "url": "https://aclanthology.org/2020.findings-emnlp.311/",
   "year": 2020
  },
  "publisher": {
   "authors": [
    "Tao Li",
    "Daniel Khashabi",
    "Tushar Khot",
    "Ashish Sabharwal",
    "Vivek Srikumar"
   ],
   "org": "BIG-bench collaboration",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/unqover"
  },
  "released": "2020-11",
  "repo_url": "https://github.com/allenai/unqover",
  "saturation": {
   "as_of": "",
   "note": "Historical transformer analyses do not establish a current ceiling.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "BIG-bench UnQover README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/unqover/README.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "UNQOVER paper",
    "url": "https://aclanthology.org/2020.findings-emnlp.311/"
   },
   {
    "accessed": "2026-09-09",
    "title": "AllenAI UnQover implementation",
    "url": "https://github.com/allenai/unqover"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench UnQover task.py",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/unqover/task.py"
   }
  ],
  "status": "active",
  "subcategory": "stereotyping bias in underspecified QA",
  "summary": "UnQover probes gender, nationality, ethnicity and religion stereotypes with underspecified span-question-answering templates.",
  "tags": [
   "big-bench",
   "safety",
   "bias",
   "question-answering",
   "social-bias"
  ],
  "task_format": "Generated span-based QA templates covering gender-occupation, nationality, ethnicity and religion, with candidate-order and polarity perturbations."
 }
}