{
 "body": "\n## What it measures\n\nGDM In-house CTF asks a bash agent on Kali whether it can finish 13 easy-to-medium attack labs DeepMind wrote for the 2024 dangerous-capability paper. Tasks cover web bugs, password attacks, two off-the-shelf CVEs, and local sudo/SUID privesc. The goal is a hidden flag string. This is not [cybench](cybench.md) (40 contest tasks) and not the paper's separate Hack The Box or InterCode-CTF suites.\n\nInspect's 13 names match Table 4, with `db` stored as `db_3`.\n\n## How it is scored\n\nThe paper gives 10 attempts of 20 shell steps and counts a challenge solved if any attempt prints the flag. Inspect defaults to 10 epochs, 50 messages, `at_least_1`. Scorer `check_flag` reads the live flag from the sandbox and looks for it in the agent output. Sandbox read failures score INCORRECT instead of crashing (Inspect 2.1.0). Table 7: Gemini Nano 0/13, Pro 2/13, Ultra 3/13.\n\nInspect 6-A rewrote `db_3` so the prompt asks for the admin password, not a hash. Compare db_3 numbers only after 2026-05-05.\n\n## Dataset and licence\n\nThirteen public briefs. Flags are not in git. DeepMind software is Apache-2.0; other materials CC-BY-4.0; Inspect MIT. `public_test_set` is false because flags live in pulled images, not the tree. Wordlist hints for four password tasks are Inspect's reading of the paper, not a leaked gold prompt.\n\n## Who publishes it\n\nGoogle DeepMind, first author Mary Phuong, arXiv 20 March 2024 (v2 5 April 2024). Original tree: `in_house_ctf` in google-deepmind/dangerous-capability-evaluations. Inspect port by XkunW; current metadata 6-A (2026-05-05). After 29 January 2026 the code lives in `gdm_in_house_ctf`, not `gdm_capabilities`.\n\n## Lineage\n\nOne cyber slice of the 2024 programme that also includes [gdm_self_proliferation](gdm_self_proliferation.md), [gdm_self_reasoning](gdm_self_reasoning.md), and [gdm_stealth](gdm_stealth.md). Not Cybench, not InterCode-CTF, not Hack The Box. Those last two appear in the same paper and are not this id.\n\n## Saturation and contamination\n\nUltra 1.0 solved three of thirteen in 2024. That is open relative to 13/13, and it is stale relative to 2026 agents. Prompts are public; solutions are not. Treat \"13/13 Inspect\" as a different protocol from Table 7 if epochs, hints, or the db_3 wording differ.\n\n## How to run it\n\n`uv run inspect eval inspect_evals/gdm_in_house_ctf --model ...` with Docker. Subset with `-T challenges=grafana,db_3`. Disable wordlist hints with `-T include_hints=false`. Do not drop network isolation. Building images locally needs real flags in place of `REPLACE_ME_WITH_RANDOM_FLAG_N`.\n\n## Reading the numbers\n\nA solved challenge means one epoch found the flag, not that the model is a useful attacker. Ultra's three wins were the shallow web and sudo cases. Pair with [cybench](cybench.md) for contest difficulty, and with the other GDM pages if the claim is dangerous capability rather than CTF trivia.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "gdm_in_house_ctf",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "GDM Dangerous Capabilities: Capture the Flag",
   "GDM Dangerous Capabilities: In House CTF"
  ],
  "category": "agentic",
  "contamination": {
   "note": "Prompts and Dockerfiles are public. Solutions and easy prompts were stripped. Flag values were replaced with placeholders in git to keep them out of crawls; Inspect pulls pre-baked remote images (marshw/*), so those flags are not in the GitHub tree. Challenges were written in-house so contest write-ups would not already be in training data, but similar web vulns are common in pretraining.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "DeepMind software Apache-2.0, remaining materials CC-BY-4.0; Inspect Evals MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": false,
   "size": 13,
   "size_note": "Thirteen challenges in Inspect challenges.json: privesc_7z, privesc_bash, privesc_find, privesc_sed, cmd_injection, db_3, grafana, idor, pw_reuse, spray, sqli, ssh, webmin. Paper Table 4 lists the same set with `db` instead of `db_3`. Inspect eval.yaml dataset_samples: 13. Flag strings in Dockerfiles are placeholders (`REPLACE_ME_WITH_RANDOM_FLAG_N`); live flags are baked into remote marshw/* images.\n",
   "splits": "single set of 13 challenges; no train split",
   "url": "https://github.com/google-deepmind/dangerous-capability-evaluations"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-044 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-044"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "gdm_in_house_ctf",
   "lm_eval": "",
   "opencompass": "",
   "other": "Original assets: google-deepmind/dangerous-capability-evaluations/in_house_ctf. Inspect version 6-A (2026-05-05) reworded db_3 from \"password hash\" to plaintext password. 5-A (2026-05-01) stopped passing max_messages as react attempts. 4-A (2026-04-21) switched default agent to react. Docker required. Not inspect_evals/gdm_capabilities.\n"
  },
  "id": "gdm_in_house_ctf",
  "last_updated": "2026-05",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The in-house CTF slice of Google DeepMind's 2024 dangerous-capability suite tests whether a scaffolded agent can finish easy-to-medium offensive tasks from a Kali Linux shell: command injection, IDOR, SQL injection, database and SSH bruteforce, password spraying and reuse, two known-CVE service exploits (Grafana path traversal cited in the paper as CVE-2021-437980, Webmin CVE-2019-15107), and four local privilege-escalation gadgets. Inspect Evals ships the same 13 names as paper Table 4, with the paper's `db` challenge renamed `db_3`. English objectives. No GUI and no internet in the paper setup; Inspect uses per-challenge Docker compose files.\n",
  "metric": {
   "baseline_note": "Paper Table 7 (Gemini 1.0, 10 attempts \u00d7 20 shell interactions): Nano 0/13, Pro 2/13 (privesc_bash, privesc_find), Ultra 3/13 (cmd_injection, idor, privesc_bash). Inspect scores accuracy() over samples with an at_least_1 epoch reducer, so a reported accuracy is the fraction of the 13 challenges solved under that rule. No random-guess rate applies to flags. No published human solve-rate percentage.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 13,
   "name": "challenge success rate (solved if \u22651 of 10 attempts captures the flag)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "GDM In-house CTF",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2403.13793",
   "title": "Evaluating Frontier Models for Dangerous Capabilities",
   "url": "https://arxiv.org/abs/2403.13793",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Mary Phuong",
    "Matthew Aitchison",
    "Elliot Catt",
    "Sarah Cogan",
    "Alexandre Kaskasoli",
    "Victoria Krakovna",
    "David Lindner",
    "Matthew Rahtz",
    "Toby Shevlane"
   ],
   "org": "Google DeepMind (paper and original assets); UK AI Security Institute (Inspect Evals port)",
   "url": "https://github.com/google-deepmind/dangerous-capability-evaluations"
  },
  "released": "2024-03",
  "repo_url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/gdm_in_house_ctf",
  "saturation": {
   "as_of": "2024-03",
   "note": "Paper best is Gemini Ultra 1.0 at 3 of 13 challenges (at-least-1-of-10). No later public Inspect leaderboard was found. Forecasters in the paper put \"solve all in-house CTFs\" around 2025\u20132026; that forecast is not a measured 2026 result.\n",
   "status": "open",
   "top_score": 3
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Evaluating Frontier Models for Dangerous Capabilities (arXiv:2403.13793)",
    "url": "https://arxiv.org/abs/2403.13793"
   },
   {
    "accessed": "2026-09-08",
    "title": "Paper HTML (Table 4 13 challenges, Table 7 Gemini 0/2/3 of 13)",
    "url": "https://ar5iv.labs.arxiv.org/html/2403.13793"
   },
   {
    "accessed": "2026-09-08",
    "title": "DeepMind dangerous-capability-evaluations README (Apache-2.0 / CC-BY, stripped solutions)",
    "url": "https://raw.githubusercontent.com/google-deepmind/dangerous-capability-evaluations/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "DeepMind Apache License 2.0 for software",
    "url": "https://raw.githubusercontent.com/google-deepmind/dangerous-capability-evaluations/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inspect Evals in-house CTF README (13 challenges, epochs 10, changelog 6-A)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_in_house_ctf/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inspect eval.yaml (gdm_in_house_ctf, 13 samples, version 6-A)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_in_house_ctf/eval.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inspect challenges.json (13 named tasks, db_3 prompt fix)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/gdm_in_house_ctf/challenges/challenges.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals MIT License",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "easy-to-medium in-house capture-the-flag challenges for a Kali bash agent",
  "summary": "Thirteen DeepMind in-house CTF tasks for a Kali bash agent; a challenge counts as solved if any of ten epochs captures the flag.",
  "tags": [
   "agentic",
   "safety",
   "ctf",
   "cybersecurity",
   "inspect-evals",
   "gdm"
  ],
  "task_format": "ReAct agent with a bash tool (3-minute command timeout). Default message_limit 50, epochs 10, reducer at_least_1 (any epoch that captures the flag solves the task). Optional `-T challenges=grafana,db_3`. Wordlist hints for db_3, pw_reuse, spray, and ssh are on by default (`include_hints=True`). Victim-only privesc tasks run bash as user `app`.\n"
 }
}