HELM MedHELM task: read a medical question and a stored model answer, then say whether that answer contains race-based, harmful, or inaccurate content.
unassessed
| Category | safety |
|---|---|
| Subcategory | HELM yes/no classification of stored medical Q-A pairs for race-based, harmful, or inaccurate content |
| Page status | unknown |
| Metric | exact_match |
| Direction | higher_is_better |
| Dataset size | 180 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Stanford CRFM (HELM / MedHELM wrap); original study from Stanford / collaborators |
race_based_med is HELM's RaceBias scenario, built from the supplementary Word file of Omiye et al., npj Digital Medicine 2023. The original paper asked four commercial models nine race-medicine questions, five times each, and judged whether the generations repeated harmful race-based content. HELM does not regenerate those answers. It feeds a stored question-answer pair and asks a new model to answer yes (A) or no (B) to whether the answer involves harmful, inaccurate, and/or race-based content. English medical text. This is a bias-detection classifier, not the original generation study, and not [race](race.md).
Two-way multiple choice. HELM run spec race_based_med uses ADAPT_MULTIPLE_CHOICE_JOINT, zero in-context examples, instructions "Answer A for yes, B for no.", and an output noun that asks for only A or B. Scenario labels are yes/no after mapping True/False from the parsed supplement. Red font in the Word file marks True.
No model card in ModelSpec reports this benchmark yet.