Meta's fourth CyberSecEval release, adding CrowdStrike-built defensive benchmarks (malware analysis, threat-intel reasoning) and an automated-patching benchmark to the prior risk and insecure-code tests.
unassessed
| Category | safety |
|---|---|
| Subcategory | LLM cybersecurity risk and capability suite: adds CrowdStrike-built defensive benchmarks (malware analysis, threat-intelligence reasoning, as CyberSOCEval) and an automated-vulnerability-patching benchmark (AutoPatchBench) to the prior version's risk and insecure-code tests |
| Page status | active |
| Metric | per-category rates and scores -- risk/insecurity rates for most inherited categories; multi-answer accuracy plus Jaccard similarity for the new CyberSOCEval tasks; patch-verification success rate for AutoPatchBench |
| Direction | lower_is_better |
| Unit | % |
| Publisher | Meta (the Purple Llama project), with CyberSOCEval developed in partnership with CrowdStrike |
CyberSecEval 4 builds on CyberSecEval 3's risk categories and adds three new benchmarks the repository documents as its headline additions: two developed with CrowdStrike under the name CyberSOCEval -- Malware Analysis (can a model correctly answer multi-answer questions about a piece of malware from its detonation report) and Threat Intelligence Reasoning (can a model extract correct answers from unstructured threat-intelligence reports, as text or as report images) -- and AutoPatchBench, which measures whether an LLM agent can generate a security patch for a real, fuzzer-discovered crash in native (C/C++) code that a verification pipeline confirms actually fixes the bug. Unlike most of the suite, the CyberSOCEval benchmarks measure a defensive capability that is good to have more of, not a risk to minimize -- the first point in the CyberSecEval series where a higher score is unambiguously the desirable direction for part of the suite. CyberSecEval 4 also carries forward CyberSecEval 3's MITRE compliance, MITRE False Refusal Rate, instruct and autocomplete insecure-code tests, and multilingual prompt injection and multi-turn phishing tests, largely unchanged.
Continuing categories (largely unchanged from CyberSecEval 3, confirmed by exact sample counts): MITRE compliance (1,000 prompts), MITRE FRR (750 prompts), instruct and autocomplete insecure-code tests (1,916 prompts each), multilingual prompt injection (1,004 cases, the same file as CyberSecEval 3), and multi-turn phishing (a subset of the CyberSecEval 3 challenge set). New: Malware Analysis (609 multiple-answer questions built from detonation reports of public malware samples targeting Windows-operated businesses, with up to 10 possible correct options per question), Threat Intelligence Reasoning (588 question-answer pairs drawn from 45 distinct threat-intelligence reports, answerable from report text, report images, or both), and AutoPatchBench (an LLM agent is given a fuzzer-discovered crash from 178 bugs across 11 crash types, drawn from the ARVO dataset of reproducible open-source vulnerabilities, and must produce a patch that a fuzzing-and-differential- testing pipeline verifies actually resolves the crash without breaking the target).
No model card in ModelSpec reports this benchmark yet.