A Chinese BIG-bench task that scores occupation-title gender bias and accuracy at inferring gender from gendered terms and names.
unassessed
| Category | safety |
|---|---|
| Subcategory | BIG-bench Chinese occupation-bias and gender-identification probes |
| Page status | unknown |
| Metric | normalized_aggregate_score (BIG-bench composite of bias, stereotype, and identification) |
| Direction | higher_is_better |
| Dataset size | 1751 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
gender_sensitivity_chinese is the Chinese counterpart of gender_sensitivity_english. It runs two zero-shot probes: a neutrality test over 827 occupation titles, and an identification test over 924 gendered terms and names. Neutrality asks whether next-token mass after a title plus 是 ("is/are") splits evenly across listed male and female words. Identification asks whether the same comparison recovers the labelled gender of a kinship term or a constructed Chinese name. There is no PTB perplexity arm and no non-binary term list, unlike the English sibling.
Programmatic next-token comparison, not a JSON multiple-choice file. Prompts are occupation or name plus 是; targets are ten male and ten female definition characters or words. Preferred scores are gender_bias_score and gender_stereotype_score on [-1, 0] (0 is unbiased) and mean_accuracy on identification. Canary GUID embedded. Zero-shot.
No model card in ModelSpec reports this benchmark yet.