{
 "body": "\n## What it measures\n\nWMDP probes whether a model can answer multiple-choice questions covering knowledge adjacent to\nbiological, chemical and cyber weapons risk, split into three subsets: WMDP-Bio, WMDP-Cyber and\nWMDP-Chem. The authors are explicit it is a proxy rather than a direct capability test -- to avoid\npublishing genuinely dangerous material, the 3,668 questions were written and reviewed by academics and\ntechnical consultants to sit at the level of precursor, neighbouring and component knowledge, each\nchecked by at least two subject-matter experts and screened for compliance with US export-control rules.\nWMDP serves two purposes: an evaluation of hazardous-adjacent knowledge a model can surface, and a\nconcrete optimisation target for machine unlearning research, which is why the paper pairs it with its\nown unlearning method (RMU, Representation Misdirection for Unlearning) rather than presenting it purely\nas a capability leaderboard.\n\n## How it is scored\n\nEvery question is four-option multiple choice, scored as plain accuracy, giving a 25% random-chance\nbaseline. This is the one entry in this repository where scoring direction needs a careful reading:\naccuracy still increases with a model's underlying knowledge, exactly as on any other multiple-choice\nbenchmark -- larger, more capable models scored higher in the paper's own baseline table. What differs is\nwhich direction is desirable. The paper states its goal plainly: reduce \"QA accuracy on WMDP while\nmaintaining performance on other benchmarks, such as MMLU.\" A low WMDP score is only meaningful as a\nsafety signal when read next to a general-capability benchmark that stayed high; a low WMDP score\nalongside a depressed MMLU score shows nothing about successful unlearning.\n\n## Dataset and licence\n\nThe dataset is 3,668 multiple-choice questions: 1,273 in WMDP-Bio, 1,987 in WMDP-Cyber and 408 in\nWMDP-Chem, each a single test split with no train or validation portion, confirmed from the official\nHugging Face dataset card's split metadata and matching the paper. The lm-evaluation-harness task README\nstates a different total, 4,157 questions (1,520/2,225/412 by subset); this page treats the paper and\ndataset-card figure as better-corroborated, since the two agree independently, and flags the mismatch\nrather than silently picking one. The dataset (`cais/wmdp`) carries an MIT licence on Hugging Face.\nQuestions were deliberately curated to exclude operationally dangerous detail even though the dataset is\nfully public.\n\n## Who publishes it\n\nWMDP was produced by a large, multi-institution author group led by Nathaniel Li, Alexander Pan and\nAnjali Gopal, with the Center for AI Safety (CAIS) as coordinating organisation and Dan Hendrycks as\nsenior author; contributors span academic groups (including UC Berkeley and MIT), biosecurity specialists\nat SecureBio, and industry participants. It was posted to arXiv in March 2024. The project is maintained\nat wmdp.ai and github.com/centerforaisafety/wmdp, alongside the RMU reference implementation.\n\n## Lineage\n\nWMDP does not sit in a benchmark family catalogued in this repository and has no confirmed predecessor or\nsuccessor. It is best understood in relation to the general-capability benchmarks it is designed to be\nread against, most notably MMLU, which the paper names directly as the companion metric that should stay\nhigh while WMDP goes down for a successful, targeted unlearning result -- the two should always be\nreported together rather than WMDP alone. It also sits within a broader line of machine-unlearning\nresearch (of which RMU is the paper's own contribution) this repository does not otherwise catalogue.\n\n## Saturation and contamination\n\nWhether WMDP is \"saturated\" depends on which direction you read it in, so this page states both rather\nthan pick one. As a raw capability ceiling, the paper's 2024 baseline table put GPT-4 at 86.2% (Bio),\n73.6% (Cyber) and 81.6% (Chem) -- well above the 25% random baseline, indicating the questions were not\nespecially obscure to a strong 2024 model. As a safety-desirable floor, the paper's own RMU method pushed\nseveral open models down toward the high 20s and low 30s percent on Bio and Cyber while mostly preserving\nMMLU. No 2025-2026 frontier-model score on unmodified WMDP was found for this page. Contamination risk is\nhigh: the full question set has been public and unchanged since March 2024 with no private holdout, so\nexposure during later training is plausible, even though questions were curated to avoid teaching\noperational detail by themselves.\n\n## How to run it\n\nlm-evaluation-harness groups the benchmark under the tag `wmdp`, covering three subtasks -- `wmdp_bio`,\n`wmdp_cyber`, `wmdp_chemistry`. inspect_evals implements the same three subsets separately (`wmdp_bio`,\n`wmdp_chem`, `wmdp_cyber`) with no combined task. The authors' own evaluation and RMU code is at\ngithub.com/centerforaisafety/wmdp. No HELM, OpenCompass or BIG-bench implementation was confirmed. Because\nthe benchmark is typically reported as three separate per-domain percentages rather than one blended\nfigure, check which subset a reported number covers before comparing it to another source.\n\n## Reading the numbers\n\nA WMDP score only means something next to a general-capability number reported alongside it: a low WMDP\nscore paired with strong MMLU is evidence of successful, targeted removal of hazardous-adjacent\nknowledge; a low WMDP score paired with a similarly depressed general benchmark just shows a weaker\nmodel, not a safer one. A high WMDP score is not, by itself, evidence a model can be used to build a\nweapon -- the question set is built deliberately around precursor and component knowledge rather than\noperational detail, so the benchmark could be published safely -- but it does indicate the model can\nsurface hazardous-adjacent information on request, the risk the benchmark is a proxy for. Treat\nper-domain scores separately rather than averaging Bio, Cyber and Chem into one number, since the domains\nprobe different threat models and models do not track uniformly across all three.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "wmdp",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Weapons of Mass Destruction Proxy Benchmark"
  ],
  "category": "safety",
  "contamination": {
   "note": "The full 3,668-question set and its answer key have been public on Hugging Face and GitHub under an MIT licence since March 2024 -- over two years before this research pass -- with no private held-out portion, so the literal questions are plausibly present in newer pretraining and fine-tuning corpora. The authors' own mitigation is different from most benchmarks': rather than holding data back, they filtered what was published so that memorising the question set would not, by itself, teach a model genuinely operational hazardous detail. That design choice reduces one kind of harm from publication but does not reduce ordinary benchmark contamination -- a model that has seen the WMDP question-answer pairs during training can still score higher on this specific test than its general hazardous-knowledge level would predict, in either direction.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 3668,
   "size_note": "3,668 multiple-choice questions, confirmed by both the paper and the official Hugging Face dataset card's per-config split sizes: 1,273 in WMDP-Bio, 1,987 in WMDP-Cyber and 408 in WMDP-Chem (all in a single test split per config, no train/validation split). The lm-evaluation-harness task README's own abstract text instead states 4,157 questions (1,520 bio / 2,225 cyber / 412 chemistry) -- a discrepancy between that harness's documentation and the paper plus official dataset card, which agree with each other; this page reports the paper/dataset-card figure as the better-corroborated one and flags rather than resolves the mismatch, since it sits in lm-evaluation-harness's own documentation rather than in this repository's data.\n",
   "splits": "single test split per subset (wmdp-bio, wmdp-cyber, wmdp-chem); no train/validation split",
   "url": "https://huggingface.co/datasets/cais/wmdp"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "wmdp_bio, wmdp_chem, wmdp_cyber (three separate tasks, no single combined task)",
   "lm_eval": "wmdp (group tag over wmdp_bio, wmdp_cyber, wmdp_chemistry subtasks)",
   "opencompass": "",
   "other": "Reference evaluation and RMU unlearning code at github.com/centerforaisafety/wmdp."
  },
  "id": "wmdp",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "WMDP measures whether a model can answer multiple-choice questions that probe hazardous knowledge adjacent to biological, cyber and chemical weapons risk, across three subsets: WMDP-Bio (enhanced potential pandemic pathogens, reverse genetics, bioweapons history, viral vectors, pathogen access), WMDP-Cyber (reconnaissance, weaponization, vulnerability discovery, exploitation, post-exploitation) and WMDP-Chem (synthesis, procurement, purification, deployment mechanisms, detection evasion). The authors describe it explicitly as a proxy, not a direct test of weapons-development capability: to avoid publishing genuinely dangerous material, questions were written and vetted by academics and technical consultants to cover precursor, neighbouring and component knowledge rather than operational detail, checked by at least two experts each, and cross-checked for compliance with US export-control rules (ITAR and EAR). WMDP serves two roles at once -- an evaluation of what hazardous-adjacent knowledge a model can produce, and a benchmark for unlearning methods that try to remove that knowledge while leaving general capability intact.\n",
  "metric": {
   "baseline_note": "This is the one benchmark in this repository's catalogue where a lower score is the desirable outcome, and that needs care to state precisely: raw accuracy still rises with a model's general knowledge and capability, the same as any other multiple-choice benchmark -- GPT-4 scored highest among models the paper tested. What is inverted is desirability, not the metric's arithmetic: WMDP exists as a target for machine unlearning, and the paper states its goal directly as reducing \"question-answer (QA) accuracy on WMDP while maintaining performance on other benchmarks, such as MMLU.\" A low WMDP score achieved by an otherwise broadly capable model (strong MMLU, weak WMDP) is the sought-after outcome; a low score from a simply weak model is not evidence of successful, targeted unlearning. Four-option multiple-choice gives a random baseline of 25%. No controlled human expert baseline is published.\n",
   "direction": "lower_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "WMDP (Weapons of Mass Destruction Proxy)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2403.03218",
   "title": "The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning",
   "url": "https://arxiv.org/abs/2403.03218",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Nathaniel Li",
    "Alexander Pan",
    "Anjali Gopal",
    "et al."
   ],
   "org": "Center for AI Safety (CAIS), with a multi-institution author group including UC Berkeley, MIT and SecureBio",
   "url": "https://www.wmdp.ai"
  },
  "released": "2024-03",
  "repo_url": "https://github.com/centerforaisafety/wmdp",
  "saturation": {
   "as_of": "",
   "note": "\"Top score\" is an ambiguous idea for WMDP given its inverted goal, so this page leaves it unset rather than pick one reading. As a raw capability ceiling, the paper's own 2024 baseline table has GPT-4 highest among tested models at 86.2% (Bio), 73.6% (Cyber) and 81.6% (Chem), with Llama 2 70B, Mixtral-8x7B and Yi-34B all well above the 25% random baseline too -- indicating the underlying knowledge was not obscure for capable 2024 models. As a safety-desirable minimum, the paper's own RMU unlearning method instead pushed several 7B-34B models down toward the mid-to-high 20s/30s percent range on Bio and Cyber while retaining most MMLU performance. No source read for this page gave a 2025-2026 frontier-model score on unmodified WMDP, so a current reading in either direction is not established here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning",
    "url": "https://arxiv.org/abs/2403.03218"
   },
   {
    "accessed": "2026-09-08",
    "title": "The WMDP Benchmark, full text (ar5iv) -- domain descriptions, dataset table, baseline and RMU results",
    "url": "https://ar5iv.labs.arxiv.org/html/2403.03218"
   },
   {
    "accessed": "2026-09-08",
    "title": "wmdp.ai -- official WMDP project page",
    "url": "https://www.wmdp.ai"
   },
   {
    "accessed": "2026-09-08",
    "title": "cais/wmdp dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/cais/wmdp"
   },
   {
    "accessed": "2026-09-08",
    "title": "cais/wmdp dataset card API -- exact per-config split sizes and MIT licence tag",
    "url": "https://huggingface.co/api/datasets/cais/wmdp"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness wmdp task README (wmdp_bio, wmdp_cyber, wmdp_chemistry)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/wmdp"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals wmdp task README",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/wmdp"
   }
  ],
  "status": "active",
  "subcategory": "hazardous-knowledge proxy measure for machine unlearning research",
  "summary": "A 3,668-question multiple-choice proxy for hazardous biosecurity, cybersecurity and chemical-security knowledge, built as a target for unlearning research; a lower score is the safety-desirable outcome.",
  "tags": [
   "safety",
   "unlearning",
   "hazardous-knowledge",
   "biosecurity",
   "cybersecurity",
   "chemical-security",
   "multiple-choice",
   "dual-use"
  ],
  "task_format": "Four-option multiple-choice questions (random chance 25%), answered zero-shot or few-shot; no free-text generation."
 }
}