Tests whether an autonomous LLM agent can actually exploit 40 real, critical-severity web application CVEs inside a live sandbox, not whether it can answer security questions.
unassessed
| Category | agentic |
|---|---|
| Subcategory | autonomous exploitation of real-world web-application CVEs in a live sandbox |
| Page status | active |
| Metric | attack success rate (fraction of the 40 CVEs where an agent achieves any one of the eight standard attack goals) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 40 |
| Dataset licence | Apache-2.0 |
| Publisher | University of Illinois Urbana-Champaign (Kang Lab) |
CVE-Bench measures whether an autonomous LLM agent can exploit real, critical-severity web application vulnerabilities inside a live sandbox, rather than whether a model can answer questions about security. Each of its 40 tasks corresponds to one real CVE from the National Vulnerability Database, hosted as a running, vulnerable web application in a set of Docker containers alongside a reference exploit the authors built to confirm the vulnerability is genuinely reachable. The agent must plan, issue tool calls, and observe results across multiple steps to achieve one of eight standardized attack goals -- this is why this page categorizes it as agentic rather than domain: scoring depends on live, multi-step interaction with a real system, not on a static answer.
An LLM agent is placed against a live, vulnerable target container and must achieve one of eight standardized attack goals (denial of service, file access, file creation, database modification, database access, unauthorized administrator login, privilege escalation, or an outbound-service request), graded automatically. Each CVE is run under two settings: zero-day (the agent gets only a task description and must find the vulnerability itself) and one-day (the agent additionally gets a high-level description of the known vulnerability). The paper reports success@1 and success@5 across five independent attempts.
No model card in ModelSpec reports this benchmark yet.