{ "body": "\n## What it measures\n\nGDM InterCode CTF asks whether a tool-using agent can solve easy picoCTF challenges inside a sandbox. Each task ships a short English brief and local files. The agent must recover a flag string of the form `picoCTF{...}`. The original InterCode paper built 100 such tasks as an interactive coding environment, with execution feedback instead of one-shot code generation. Google DeepMind reused the suite in 2024 as a cyber dangerous-capability probe on Gemini 1.0. Inspect Evals is the public runnable port. It is not [Cybench](cybench.md), and it is not DeepMind's separate in-house or Hack The Box CTFs from the same paper.\n\n## How it is scored\n\nInspect scores each of 78 tasks as success if the submitted text includes the gold flag (`includes()`). The headline figure is mean accuracy on that set. Defaults are three submit attempts and 50 messages. DeepMind scored 81 tasks after dropping 19 internet challenges, and reported category fractions for Gemini Nano, Pro, and Ultra (Table 6: 6/81, 22/81, 24/81). Those 81-task rates are not comparable to Inspect's 78-task table without mapping ids. Inspect's February 2026 report used one epoch and `react()` (since changelog 4-A, 2026-04-21), not DeepMind's bash-only shell.\n\n## Dataset and licence\n\nInterCode-CTF has 100 picoCTF items. Inspect downloads `princeton-nlp/intercode` at commit `c3e46d8` and drops 22 ids that need the network, leaving 78. DeepMind's footnote lists 19 omitted internet ids; Table 6's 81 matches 100 minus those 19. Inspect's README also says DeepMind dropped seven extra non-internet tasks; that 26-count does not match Table 6. InterCode and inspect_evals are MIT. picoCTF challenge text is not given a separate licence in the files opened here. Solutions ship in `task_assets/*/solution/README.md` and are copied into sample metadata.\n\n## Who publishes it\n\nJohn Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao released InterCode in June 2023 (arXiv:2306.14898, v3 October 2023). DeepMind's dangerous-capability paper (Phuong et al., March 2024) is the GDM protocol. UK AISI hosts the Inspect task (`gdm_intercode_ctf`), contributed by jjallaire. Folder split from a shared GDM evals directory in inspect_evals 2.0.0 (2026-01-29). No public live leaderboard was opened.\n\n## Lineage\n\nThis page is the Inspect spelling `gdm_intercode_ctf`, not a second InterCode-CTF id. InterCode also ships Bash, SQL, and Python environments; those are not this page. DeepMind's 2024 paper adds other cyber suites (in-house CTF, Hack The Box) that Inspect does not run here. [Cybench](cybench.md) is a later, harder 40-task professional CTF eval.\n\n## Saturation and contamination\n\nInspect's 2026 Claude Opus 4.5 run sits at 0.936 on 78 tasks, so the easy picoCTF slice no longer separates frontier agents well. Gemini Ultra's 24/81 in 2024 is a different set and harness. Assets, flags, and written solutions are public, so contamination risk is high. Compare only matching exclusion lists and agent stacks.\n\n## How to run it\n\n`uv run inspect eval inspect_evals/gdm_intercode_ctf --model ...` after Docker is installed. Optional `-T` flags: `shuffle`, `max_attempts`, `max_messages`, `sample_ids`, package lists, `sandbox_config`. Changelog 4-B (2026-06-17) added k8s-style sandbox specs. The bundled image is generated from `Dockerfile.template`. Do not compare bash-only DeepMind traces with Inspect's bash-plus-python `react()` agent.\n\n## Reading the numbers\n\nA 0.9 Inspect score means the agent submitted the right flag on most of 78 offline picoCTF tasks, not that it can do live professional CTF. Quote the task count (78 vs 81 vs 100), the excluded ids, and whether tools include Python. `includes()` can credit a flag buried in extra text. Solutions in the download make memorisation hard to rule out. Pair this number with a harder suite such as [Cybench](cybench.md) before talking about cyber capability.\n", "build": { "built_at": "2026-09-09T16:56:50+00:00", "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11", "eligibility_as_of": "2026-09-09" }, "disposition": { "canonical_id": "gdm_intercode_ctf", "reasons": [], "status": "unassessed", "verified_results": [] }, "models_covered": [], "page": { "aliases": [ "InterCode-CTF", "InterCode CTF", "inspect_evals/gdm_intercode_ctf" ], "category": "agentic", "contamination": { "note": "picoCTF tasks and InterCode assets are public. Inspect's loader also reads each task's solution/README.md into sample metadata. Flags follow a known picoCTF{...} pattern. No source opened here measured training overlap.\n", "risk": "high" }, "dataset": { "languages": [ "en" ], "license": "MIT", "modalities": [ "text", "code" ], "public_test_set": true, "size": 78, "size_note": "InterCode-CTF is 100 picoCTF tasks (Yang et al. 2023). Inspect filters 22 ids that need the internet: 1, 7, 9, 14, 16, 20, 28, 29, 35, 39, 41, 42, 43, 54, 57, 62, 66, 73, 87, 88, 89, 95 (eval.yaml dataset_samples: 78). DeepMind's main text omits 19 internet tasks (footnote list matches 19 of those ids) and scores 81 remaining in Table 6. Inspect README also claims DeepMind dropped seven extra non-internet ids; that 26-task story does not match Table 6's 81. Data are loaded from princeton-nlp/intercode at commit c3e46d827cfc9d4c704ec078f7abf9f41e3191d8 (ic_ctf.json plus task_assets).\n", "splits": "single evaluation set; no train split. Inspect pins a GitHub zip of the CTF folder.", "url": "https://github.com/princeton-nlp/intercode" }, "freshness": { "researched": "2026-09-08", "researched_by": "Grok Build, batch-045 (Codex coordinated)", "reviewed": "2026-09-08", "reviewed_by": "Grok Build independent review, batch-045" }, "harness": { "bigbench": "", "helm": "", "inspect_evals": "gdm_intercode_ctf", "lm_eval": "", "opencompass": "", "other": "inspect eval inspect_evals/gdm_intercode_ctf. eval.yaml version 4-B (2026-06-17). Changelog 4-A (2026-04-21) replaced basic_agent() with react(). Needs Docker. Not Cybench and not DeepMind's separate in-house or Hack The Box CTFs.\n" }, "id": "gdm_intercode_ctf", "last_updated": "2026-06", "leaderboard_url": "", "lineage": { "family": "", "predecessor": "", "successors": [], "variants": [] }, "measures": "The agent is a capture-the-flag player on Ubuntu. Each item is a picoCTF challenge from InterCode-CTF: a natural-language brief, local assets, and a hidden flag of the form picoCTF{...}. Categories include general Linux skill, reverse engineering, cryptography, forensics, binary exploitation, and web exploitation. Success is finding the flag with bash and python tools, not answering a static quiz. English prompts. Inspect Evals runs 78 of the original 100 tasks (22 internet-requiring tasks are dropped). DeepMind used the same suite in its 2024 dangerous-capability cyber evals, with a different exclusion list.\n", "metric": { "baseline_note": "Binary per-task success (flag found or not). Inspect README (evaluation version 2.1.0, February 2026) reports claude-opus-4-5-20251101 at 0.936 (stderr 0.028) and gpt-5.1-2025-11-13 at 0.897 (stderr 0.035) on the 78-sample set, one epoch. DeepMind Table 6 on 81 tasks: Gemini Nano 6/81, Pro 22/81, Ultra 24/81.\n", "direction": "higher_is_better", "human_baseline": null, "max_score": 1.0, "name": "accuracy", "random_baseline": null, "unit": "" }, "name": "GDM InterCode CTF", "page_kind": "benchmark", "paper": { "arxiv": "2306.14898", "title": "InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback", "url": "https://arxiv.org/abs/2306.14898", "year": 2023 }, "publisher": { "authors": [ "John Yang", "Akshara Prabhakar", "Karthik Narasimhan", "Shunyu Yao" ], "org": "Princeton NLP (InterCode); Google DeepMind (2024 protocol); UK AI Security Institute (Inspect Evals port)", "url": "https://intercode-benchmark.github.io/#ctf" }, "released": "2023-06", "repo_url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_intercode_ctf", "saturation": { "as_of": "2026-02", "note": "Inspect README, 78 tasks, one epoch, version 2.1.0 (February 2026): Claude Opus 4.5 at 0.936. DeepMind Gemini Ultra 1.0 was 24/81. The two denominators are not the same set.\n", "status": "watch", "top_score": 0.936 }, "sources": [ { "accessed": "2026-09-08", "title": "Inspect Evals gdm_intercode_ctf README (78 tasks, GDM exclusion table, Feb 2026 scores)", "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_intercode_ctf/README.md" }, { "accessed": "2026-09-08", "title": "eval.yaml (task gdm_intercode_ctf, 78 samples, version 4-B, arXiv 2306.14898)", "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_intercode_ctf/eval.yaml" }, { "accessed": "2026-09-08", "title": "dataset.py (pinned intercode commit, 22 excluded ids, prompt and solution loader)", "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_intercode_ctf/dataset.py" }, { "accessed": "2026-09-08", "title": "Task constructor (react agent, includes() scorer, Docker compose sandbox)", "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_intercode_ctf/gdm_intercode_ctf.py" }, { "accessed": "2026-09-08", "title": "InterCode paper (arXiv:2306.14898); 100 picoCTF InterCode-CTF tasks", "url": "https://arxiv.org/abs/2306.14898" }, { "accessed": "2026-09-08", "title": "InterCode HTML (CTF as