HELM scenario for DecodingTrust fairness: yes/no Adult income prediction, scored for accuracy and group gaps.
unassessed
| Category | safety |
|---|---|
| Subcategory | HELM wrap of DecodingTrust Adult income fairness (yes/no) |
| Page status | unknown |
| Metric | quasi_exact_match (headline); demographic_parity_difference; equalized_odds_difference |
| Direction | higher_is_better |
| Dataset size | 200 |
| Dataset licence | CC-BY-SA-4.0 |
| Publisher | DecodingTrust authors (UIUC / Stanford / collaborators); HELM wrap by Stanford CRFM |
decodingtrust_fairness is HELM's wrap of DecodingTrust section 10, limited to the Adult income task. Each item is an English paragraph of census-style attributes (age, education, sex, race, hours, and the rest of Adult's 14 fields; the jsonl templates say 1996). The model must answer whether annual income exceeds $50k with exactly yes or no. The paper's fairness section reports Adult only and varies sensitive attributes (sex, race) and few-shot base-rate parity. Hugging Face also lists unused Crime jsonl files. HELM's scenario only implements task=predict_annual_income (Adult files). HELM's FairnessMetric then estimates demographic parity difference and equalized odds difference using a "sex Female" substring.
HELM instruct generation, max_tokens 16, temperature 0, max_train_instances 0. Parameters: task, train_base_rate, test_base_rate, num_train, num_test, which select one jsonl filename.
No model card in ModelSpec reports this benchmark yet.