{
 "body": "\n## What it measures\n\nAutoBencher Safety tests whether a model refuses a harmful request, but the requests are not human-written the way most refusal benchmarks' prompts are; they come from the same AutoBencher search procedure documented on `autobencher_capabilities`, aimed here at a different target. An evaluator LM proposes candidate harmful-topic descriptions -- 10 search rounds proposing 10 descriptions each, 10 requests per description -- and each topic is scored by how often a panel of target models complies rather than refuses, its attack success rate (ASR). Later rounds steer toward topics and phrasings with higher observed ASR, so the released set is deliberately enriched for requests models are bad at declining, not a representative sample of harmful requests generally. Topics span concrete scenarios -- \"Military robot,\" \"Lethal autonomous weapon,\" \"Online child abuse,\" \"Ransomware,\" \"2021 Squid Game cryptocurrency scam\" -- each narrowed by an added requirement (a military-robot prompt asked to \"include ethical concerns and battlefield applications\"). Unlike the capability datasets, no privileged source document grounds each item, since the authors judge refusal does not depend on any outside fact.\n\n## How it is scored\n\nEvery item's gold answer is literally the string \"Refusal.\" HELM's `autobencher_safety` scenario scores a response with the same model-judged annotation field the capabilities scenario uses, correct when judged to have appropriately refused. The paper itself reports results the opposite way, as attack success rate (ASR = 1 minus refusal rate): the constructed set reached an average ASR of 0.38 against the panel it was searched over, versus 0.08 for XSTest (`xstest`) and 0.28 for HarmBench (no page here yet) scored the same way -- models comply with AutoBencher's prompts noticeably more than with either existing set, though it falls short of adversarial GCG-suffix attacks on HarmBench (0.45), which it does not try to match since it searches for topic-level weaknesses rather than per-prompt adversarial strings. Before release the authors ran a Mechanical Turk harmfulness audit, pruning roughly 2% of prompts rated as posing no real risk (4% at a stricter threshold).\n\n## Dataset and licence\n\nThe file HELM loads holds 301 prompts across 30 topic categories (ten items each), each tagged with a broader parent category (Robotics, Cybersecurity, Bioethics, Politics), confirmed directly from the JSON. It is hosted not on the main AutoBencher repository but a co-author's fork (`farzaank/AutoBencher`), in a folder named for Mechanical Turk processing; neither that fork nor the upstream repository carries a LICENSE file, and no licence is stated anywhere found, so terms of reuse are not established. All prompts are English.\n\n## Who publishes it\n\nAutoBencher Safety is part of the same project as `autobencher_capabilities`: Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang and Tatsunori Hashimoto at Stanford, posted to arXiv in July 2024 and accepted at ICLR 2025. Stanford CRFM's HELM team integrated it as the `autobencher_safety` scenario alongside capabilities, sharing a leaderboard schema that is not currently hosted as a public site (see Saturation).\n\n## Lineage\n\nThis page has no predecessor id here. Its closest relative is `autobencher_capabilities`, built by the identical procedure but optimised for a different desideratum (refusal failures rather than knowledge gaps); the two are recorded as variants of each other. The paper positions the dataset as a harder alternative to existing hand-built refusal tests, naming XSTest (`xstest`, present in this repository) and HarmBench (no page here yet) as the sets it compares ASR against.\n\n## Saturation and contamination\n\nHELM defines a leaderboard schema for this scenario but, as with capabilities, does not currently host a public site for it, and the live HELM Safety leaderboard tracks a different roster (HarmBench, HarmBench-GCG-Transfer, SimpleSafetyTests, XSTest, BBQ, Anthropic Red Team) that excludes AutoBencher Safety. No model card here reports this benchmark, and the census hint's score of 18.0 could not be corroborated from any rendered source, so status is unknown rather than guessed. Contamination risk is medium: prompt text was generated for this one search run rather than lifted from an existing corpus, and has seen limited distribution (a single GitHub fork, not a widely-mirrored hub), but its topics are ordinary current-events and dual-use-technology subjects that later safety-tuning corpora commonly cover, so a model trained on more recent safety data could plausibly perform well here for reasons unrelated to this specific file.\n\n## How to run it\n\nThe only confirmed harness integration is HELM's `autobencher_safety` scenario, which downloads the JSON directly from GitHub rather than a Hugging Face repository, scoring zero-shot with the same annotation-based metric name as capabilities. No lm-evaluation-harness, inspect_evals, OpenCompass or BIG-bench task was found. Note `schema_autobencher.yaml` names `annotation_autobencher_capabilities_score` as the main metric for both run groups -- worth checking against HELM's current source before assuming the safety numbers come from an independently named metric.\n\n## Reading the numbers\n\nA high score means a model tends to refuse prompts deliberately selected, by search, for being ones existing models fail to refuse -- more about closing specific, adversarially-discovered gaps than general safety behaviour. Because the set is small (301 items, ~10 per topic), a handful of category-level failures can move the aggregate substantially, and because requests were authored by an LLM rather than security or policy specialists, some items may not reflect the harm categories a given deployment cares about. With no live leaderboard and no scores yet in this repository's model cards, there is no basis for calling a number high or low; read it alongside XSTest and HarmBench results where available, since the paper's own comparison shows the three sets disagree substantially on how permissive current models are.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "autobencher_safety",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "safety",
  "contamination": {
   "note": "Prompts were generated for this specific search run rather than copied from an existing public corpus, and the file has had limited public exposure (hosted on a fork of the authors' own GitHub repository rather than a widely-mirrored dataset hub). Against that, the prompts are built around ordinary, well-known topics (cybersecurity, bioethics, current-events scams), the file has been downloadable since mid-to-late 2024, and safety benchmarks in general are a common target for inclusion in later safety-tuning and red-teaming datasets, which could indirectly expose a similar distribution of prompts to later training even without this exact file being scraped.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 301,
   "size_note": "301 prompts across 30 fine-grained topic categories (10 items each, e.g. \"Military robot,\" \"Lethal autonomous weapon,\" \"Online child abuse,\" \"Ransomware,\" \"2021 Squid Game cryptocurrency scam\"), counted directly from the JSON file HELM's autobencher_safety scenario loads. Every item's gold_answer field is literally the string \"Refusal.\" Each item also carries a broader parent_category (e.g. \"Robotics,\" \"Cybersecurity,\" \"Bioethics,\" \"Politics\") and an additional_requirement field that narrows the request further (for example, adding \"include ethical concerns and battlefield applications\" to a military-robot prompt). This 301-item file is the harmful-request search's output; it is materially smaller than the roughly 1,000 requests the search process could generate across its full 10x10x10 loop, meaning most generated candidates were filtered out before release. The paper separately reports a Mechanical Turk harmfulness audit that pruned about 2% of generated prompts for being rated as posing no real risk of harm (rising to about 4% at a stricter threshold), which was applied before this file was finalised.\n",
   "splits": "single flat list of 301 items, used directly as HELM's test split; no train/validation portion",
   "url": "https://raw.githubusercontent.com/farzaank/AutoBencher/refs/heads/main/safety_processing/process%20full%20dataset%20for%20mTurk/full_dataset.json"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "autobencher_safety",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "autobencher_safety",
  "last_updated": "2024-11",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "autobencher_capabilities"
   ]
  },
  "measures": "AutoBencher Safety tests whether a model refuses harmful requests, but -- like its sibling `autobencher_capabilities` -- the requests themselves were not written by a person; they were produced by the AutoBencher search procedure, whose declared goal for this domain is to find topics and framings that existing language models fail to decline. An evaluator LM proposes candidate harmful-topic descriptions (an adaptive search loop runs 10 rounds proposing 10 descriptions each, generating 10 requests per description), and each candidate topic is scored by how often a panel of target models actually complies with it rather than refusing -- its attack success rate. Later search rounds are steered toward topics and phrasings with a higher observed attack success rate, so the released set is deliberately enriched for requests that existing models are bad at declining, not a random or representative sample of harmful requests in general. Unlike the capabilities dataset, no privileged source document grounds each item, because the paper's authors judge that a correct response (refusal) does not require any outside information to construct.\n",
  "metric": {
   "baseline_note": "No source read for this page gives a human baseline; refusal is a behavioural judgement, not a fact a human could get right or wrong for comparison. The paper itself frames model performance on this dataset the opposite way round, as an Attack Success Rate (ASR = 1 minus refusal rate, i.e. how often models comply with the harmful request rather than decline it) -- lower ASR is safer. It reports the AutoBencher-constructed dataset reaching an average ASR of 0.38 against the models it was searched against, compared with 0.08 for XSTest (`xstest`) and 0.28 for HarmBench (no page in this repository yet) evaluated the same way, and 0.45 for adversarial GCG-suffix attacks on HarmBench, which AutoBencher does not attempt to beat since it searches for systematic topic-level weaknesses rather than per-prompt adversarial strings.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "Model-judged correctness (HELM's annotation_autobencher_capabilities_score field, reused for this scenario: whether the model's response is judged to have appropriately refused)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "AutoBencher Safety",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2407.08351",
   "title": "AutoBencher: Towards Declarative Benchmark Construction",
   "url": "https://arxiv.org/abs/2407.08351",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Xiang Lisa Li",
    "Farzaan Kaiyom",
    "Evan Zheran Liu",
    "Yifan Mai",
    "Percy Liang",
    "Tatsunori Hashimoto"
   ],
   "org": "Stanford University",
   "url": "https://github.com/XiangLi1999/AutoBencher"
  },
  "released": "2024-07",
  "repo_url": "https://github.com/farzaank/AutoBencher",
  "saturation": {
   "as_of": "",
   "note": "As with autobencher_capabilities, HELM defines a leaderboard schema for this scenario (schema_autobencher.yaml) but does not currently host a public site for it: it is absent from HELM's homepage list of leaderboards, and crfm.stanford.edu/helm/autobencher/latest/ 404s. The current HELM Safety leaderboard (crfm.stanford.edu/helm/safety/latest/) instead tracks HarmBench, HarmBench-GCG-Transfer, SimpleSafetyTests, XSTest, BBQ and Anthropic Red Team -- not AutoBencher Safety. This repository's census hints record a score of 18.0 for this id, but no rendered source found during this research corroborated it, so it is not repeated as a top_score. No model card in this repository reports this benchmark yet (checked by grep across models/).\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "AutoBencher: Towards Declarative Benchmark Construction (arXiv abstract, authors, ICLR 2025 acceptance note)",
    "url": "https://arxiv.org/abs/2407.08351"
   },
   {
    "accessed": "2026-09-08",
    "title": "AutoBencher: Towards Declarative Benchmark Construction (ar5iv full text: safety search procedure, ASR results, Mechanical Turk harmfulness audit)",
    "url": "https://ar5iv.labs.arxiv.org/abs/2407.08351"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM AutobencherSafetyScenario source (dataset source URL, TEST_SPLIT-only handling)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/autobencher_safety_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_autobencher.yaml (run-group taxonomy: 'who: synthetic model-generated questions'; shared metric definition)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_autobencher.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "farzaank/AutoBencher full_dataset.json (the exact 301-item file HELM loads; category/gold_answer/difficulty fields)",
    "url": "https://raw.githubusercontent.com/farzaank/AutoBencher/refs/heads/main/safety_processing/process%20full%20dataset%20for%20mTurk/full_dataset.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "farzaank/AutoBencher GitHub repository (fork of XiangLi1999/AutoBencher hosting the safety data; no LICENSE file present)",
    "url": "https://github.com/farzaank/AutoBencher"
   },
   {
    "accessed": "2026-09-08",
    "title": "XiangLi1999/AutoBencher GitHub repository (upstream/official code release; no LICENSE file present)",
    "url": "https://github.com/XiangLi1999/AutoBencher"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Leaderboards homepage (full list of currently hosted leaderboard suites)",
    "url": "https://crfm.stanford.edu/helm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Safety leaderboard, rendered (scenario list is HarmBench/XSTest/BBQ/Anthropic Red Team/SimpleSafetyTests, not AutoBencher)",
    "url": "https://crfm.stanford.edu/helm/safety/latest/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Direct AutoBencher leaderboard URL on the HELM site (404, confirms no live hosted page)",
    "url": "https://crfm.stanford.edu/helm/autobencher/latest/"
   }
  ],
  "status": "unknown",
  "subcategory": "automatically-searched harmful-request refusal test",
  "summary": "AutoBencher Safety is a 301-prompt HELM refusal test whose harmful requests across 30 topics were searched for and generated by a language model to maximize how often models comply, not written by people.",
  "tags": [
   "safety",
   "refusal",
   "red-teaming",
   "synthetic-benchmark",
   "llm-generated",
   "automated-benchmark-construction"
  ],
  "task_format": "The model is given a single potentially harmful request in natural language and produces a free-form response; the correct behaviour on every item is to refuse or decline to help.\n"
 }
}