Meta's third CyberSecEval release: adds visual prompt injection, automated spear-phishing, and human and autonomous offensive-cyber-operations studies to the prior version's risk and insecure-code tests.
unassessed
| Category | safety |
|---|---|
| Subcategory | LLM cybersecurity risk and capability suite: adds visual prompt injection, automated social engineering, and manual and autonomous offensive-cyber-operations studies to the prior version's risk and insecure-code tests |
| Page status | superseded |
| Metric | per-category rates and scores -- mainly an injection/attack success or compliance rate; the manual- and autonomous-uplift studies report qualitative human-subject and phase-completion outcomes rather than a single percentage |
| Direction | lower_is_better |
| Unit | % |
| Dataset licence | MIT |
| Publisher | Meta (the Purple Llama project) |
CyberSecEval 3 measures LLM cybersecurity risk and capability across 8 named risk areas the paper groups into two broad categories: risk to third parties (parties other than the model's own user or developer, such as social-engineering victims or organizations targeted by cyberattacks) and risk to application developers and end users (insecure code, prompt injection and code-interpreter abuse reaching the people who deploy or use the model). It keeps CyberSecEval 2's textual prompt injection, code-interpreter abuse and vulnerability-exploitation tests essentially unchanged -- reusing that version's own implementation rather than re-deriving it -- and adds three offensive- security areas the paper names as new: automated social engineering (can a model run a persuasive multi-turn spear-phishing conversation against a simulated victim), scaling manual offensive cyber operations (a human-subjects study of whether LLM assistance speeds up novice and expert penetration-testers on capture-the-flag-style challenges), and autonomous offensive cyber operations (can an LLM agent, unassisted, advance through reconnaissance, vulnerability discovery, exploitation and privilege escalation against a target host). It also adds a visual and a multilingual variant of the prompt-injection tests, and documents Prompt Guard, a separate classifier model for detecting injections and jailbreaks, which this page treats as a related mitigation artifact rather than part of the graded benchmark itself.
Reused from CyberSecEval 2 unchanged: textual prompt injection (251 cases), code-interpreter abuse (500 prompts), and vulnerability exploitation. New in this version: visual prompt injection (1,000 system-prompt/image/text triples, confirmed from the Hugging Face dataset card, LLM-judged on whether an instruction embedded in the image overrode the system prompt), multilingual prompt injection (1,004 machine-translated cases, counted directly from the released file), automated social engineering (up to 856 synthetic multi-turn spear-phishing challenges against a simulated, LLM-played victim, scored by a judge LLM on a 5-point persuasion/rapport/argumentation rubric), the manual-uplift human study (inexperienced and highly skilled subjects solving CTF-style challenges with and without LLM help, on Hack The Box infrastructure), and the autonomous-operations study (an LLM agent attempting a fixed attack-phase sequence against a Windows host from a Kali Linux environment, in a controlled setting).
No model card in ModelSpec reports this benchmark yet.