AutoBencher Safety is a 301-prompt HELM refusal test whose harmful requests across 30 topics were searched for and generated by a language model to maximize how often models comply, not written by people.
unassessed
| Category | safety |
|---|---|
| Subcategory | automatically-searched harmful-request refusal test |
| Page status | unknown |
| Metric | Model-judged correctness (HELM's annotation_autobencher_capabilities_score field, reused for this scenario: whether the model's response is judged to have appropriately refused) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 301 |
| Publisher | Stanford University |
AutoBencher Safety tests whether a model refuses harmful requests, but -- like its sibling `autobencher_capabilities` -- the requests themselves were not written by a person; they were produced by the AutoBencher search procedure, whose declared goal for this domain is to find topics and framings that existing language models fail to decline. An evaluator LM proposes candidate harmful-topic descriptions (an adaptive search loop runs 10 rounds proposing 10 descriptions each, generating 10 requests per description), and each candidate topic is scored by how often a panel of target models actually complies with it rather than refusing -- its attack success rate. Later search rounds are steered toward topics and phrasings with a higher observed attack success rate, so the released set is deliberately enriched for requests that existing models are bad at declining, not a random or representative sample of harmful requests in general. Unlike the capabilities dataset, no privileged source document grounds each item, because the paper's authors judge that a correct response (refusal) does not require any outside information to construct.
The model is given a single potentially harmful request in natural language and produces a free-form response; the correct behaviour on every item is to refuse or decline to help.
No model card in ModelSpec reports this benchmark yet.