Tests whether an autonomous LLM agent can solve 40 professional-level capture-the-flag cybersecurity tasks in a live sandbox, with 17 of them broken into guided subtasks for partial credit.
unassessed
| Category | agentic |
|---|---|
| Subcategory | autonomous capture-the-flag (CTF) cybersecurity task-solving, with a graded subtask mode |
| Page status | active |
| Metric | task success rate (fraction of the 40 CTF tasks solved), reported separately per scoring mode |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 40 |
| Dataset licence | Apache-2.0 (repository code); the redistributed CTF challenge content itself is not uniformly licensed -- see Dataset and licence |
| Publisher | Stanford University (Center for AI Safety collaborators Dan Boneh, Daniel E. Ho and Percy Liang among the senior authors), with a 27-author team |
Cybench measures whether an autonomous LLM agent, given shell and Python tool access inside a Kali Linux sandbox, can solve real capture-the-flag (CTF) security challenges drawn from four professional-level competitions (HackTheBox Cyber Apocalypse 2024, SekaiCTF, Glacier and HKCert). Tasks span six domains -- cryptography, web security, reverse engineering, forensics, exploitation (pwn) and miscellaneous -- and the paper anchors difficulty in human terms: the times competitive human teams took to first-solve each task ranged from a few minutes up to nearly 25 hours. This page categorizes Cybench as agentic rather than domain knowledge, because success depends on multi-step planning and tool use against a live target, not a static answer.
An agent is placed in a Kali Linux Docker sandbox with bash and code-execution tools and must find and submit a flag string for each of 40 tasks. 17 of the 40 tasks additionally ship a sequence of subtasks -- guided intermediate questions with their own answers -- scored sequentially for gradated credit. Three scoring modes result: unguided performance (binary success on the full task with no hints), subtask-guided performance (binary success on only the final subtask, i.e. full guidance up to the last step), and subtask performance (the fraction of a task's subtasks solved).
No model card in ModelSpec reports this benchmark yet.