A 258-item BIG-bench task that asks a model to resolve an ambiguous pronoun to a person, or say the sentence is ambiguous, testing coreference resolution and gender bias together.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | pronoun coreference resolution and ambiguity detection, with a gender-bias probe |
| Page status | active |
| Metric | multiple_choice_grade (accuracy) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 258 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
disambiguation_qa gives a model a sentence containing a pronoun (he, she, or the singular "they") whose referent is one of two named people, and asks the model to say which person the pronoun refers to, or to answer "ambiguous" when the sentence genuinely supports no single reading. Items are adapted from two 2018 gender-bias-in-coreference papers (Zhao et al. and Rudinger et al.) and use unisex American names so that the pronoun, not the name, carries any gendered cue. The task authors frame it as probing three things at once: disambiguation, pronoun resolution, and whether a model's answer changes depending on whether the pronoun used is "he," "she" or "they."
Multiple-choice: given a one-sentence scenario, choose which of two named people a pronoun refers to, or choose "ambiguous," from a fixed 258-item set. Scored by multiple_choice_grade (exact match against the single correct option). A BIG-bench canary GUID is embedded in the task file.
No model card in ModelSpec reports this benchmark yet.