An English BIG-bench task that scores occupation-title gender bias, gender identification from names and terms, and PTB perplexity.
unassessed
| Category | safety |
|---|---|
| Subcategory | BIG-bench English occupation-bias, gender-identification, and PTB probes |
| Page status | unknown |
| Metric | normalized_aggregate_score (BIG-bench composite of bias, stereotype, identification, PTB) |
| Direction | higher_is_better |
| Dataset size | 2035 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
gender_sensitivity_english runs three zero-shot tests. Neutrality measures whether next-token probabilities after a gender-neutral occupation plus "is" favour male, female, or listed non-binary terms, and whether occupation distributions given those terms differ (stereotype). Identification measures whether the same comparison recovers labelled gender from gendered nouns and SSA-filtered given names. A third arm reports character-level negative perplexity on a Penn Treebank string shipped in test_data.json, because de-biasing can hurt language modelling.
Programmatic next-token comparison. Neutrality and identification prompts end in " is ". Preferred scores: gender_bias_score, gender_minority_bias_score, gender_stereotype_score and gender_minority_stereotype_score on [-1, 0]; mean_accuracy on identification; negative_perplexity on PTB. Canary GUID embedded. Zero-shot.
No model card in ModelSpec reports this benchmark yet.