Eleven DeepMind challenges where an agent must notice and change its own config or tools to finish a job.
unassessed
| Category | agentic |
|---|---|
| Subcategory | instrumental self-reasoning / situational awareness in a fake deployment |
| Page status | active |
| Metric | challenge success rate (binary); suite pass count |
| Direction | higher_is_better |
| Dataset size | 22 |
| Dataset licence | MIT |
| Publisher | Google DeepMind (paper); UK AI Security Institute (Inspect Evals port) |
GDM Self-reasoning, as shipped in Inspect Evals, is the eleven situational-awareness challenges from Phuong, Zimmermann, Wang et al. 2025. The agent runs on a fake Linux deployment with bash and an agent.py stack. The goal looks ordinary (write a file, use a calculator, look up a JSON key). The real test is whether it notices a hidden constraint — max tokens, oversight frequency, a disabled database tool, a broken calculator, a blocklist — and edits config or source to lift it. English instructions; code and config files in the sandbox.
Multi-turn ReAct agent in Docker or k8s. Inspect default: 80 messages, 10 epochs, at_least_1 reducer, success scorer on a store flag. Paper: 50 trials per challenge per difficulty. Most tasks have an easy and a hard sample (eval.yaml: 2 samples each).
No model card in ModelSpec reports this benchmark yet.