1,507 real patched vulnerabilities across 188 projects; agents must write a PoC that crashes the unpatched binary but not the fix.
unassessed
| Category | agentic |
|---|---|
| Subcategory | proof-of-concept generation for historical OSS-Fuzz/ARVO vulnerabilities |
| Page status | active |
| Metric | reproduction rate (PoC crashes unpatched and not patched); new-vulnerability rate (PoC also crashes the patch) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1507 |
| Dataset licence | Apache-2.0 (official repository); bundled project trees keep their upstream licences |
| Publisher | University of California, Berkeley |
CyberGym asks an agent to reproduce a historical memory-safety bug. The default (level-1) setting gives the unpatched codebase plus a short vulnerability description. The agent must emit a proof-of-concept file that crashes the pre-patch build under sanitizers and does not crash the post-patch build. A PoC that also crashes the patched program is counted as a new vulnerability, not a reproduction. This is live tool use against compiled targets, not a quiz about CVEs.
Docker sandbox with a solver container and separate vulnerable and fixed program containers. The agent writes a PoC and submits it through submit.sh. inspect_evals defaults to a react solver, max_attempts=3, level1 only, no k8s. Four difficulty levels change which files are mounted (code only, plus description, plus crash log, plus patch and fixed tree).
No model card in ModelSpec reports this benchmark yet.