78 picoCTF tasks from InterCode-CTF: a tool-using agent must recover picoCTF{...} flags in Docker, as used in DeepMind's 2024 cyber evals.
unassessed
| Category | agentic |
|---|---|
| Subcategory | picoCTF capture-the-flag in a Docker sandbox (Inspect GDM port) |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Dataset size | 78 |
| Dataset licence | MIT |
| Publisher | Princeton NLP (InterCode); Google DeepMind (2024 protocol); UK AI Security Institute (Inspect Evals port) |
The agent is a capture-the-flag player on Ubuntu. Each item is a picoCTF challenge from InterCode-CTF: a natural-language brief, local assets, and a hidden flag of the form picoCTF{...}. Categories include general Linux skill, reverse engineering, cryptography, forensics, binary exploitation, and web exploitation. Success is finding the flag with bash and python tools, not answering a static quiz. English prompts. Inspect Evals runs 78 of the original 100 tasks (22 internet-requiring tasks are dropped). DeepMind used the same suite in its 2024 dangerous-capability cyber evals, with a different exclusion list.
Multi-turn ReAct agent in Docker. Default solver is inspect_ai.agent.react with bash and python (180s timeouts) and a submit tool. Defaults: max_attempts 3, max_messages 50. Scorer is includes() on the gold flag string. Task-level sandbox via bundled compose, or a caller-supplied SandboxEnvironmentSpec.
No model card in ModelSpec reports this benchmark yet.