Inspect Evals CodeIPI: 45 Docker coding-agent tasks that hide prompt injections in issues, comments, READMEs or configs, scored for resistance and bug-fix success.
unassessed
| Category | safety |
|---|---|
| Subcategory | indirect prompt injection on coding agents that must still fix a bug |
| Page status | active |
| Metric | accuracy (resist injection AND complete the bug fix); also IRR, TCR, detection_rate, FPR |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 45 |
| Dataset licence | MIT |
| Publisher | UK AI Security Institute (Inspect Evals) |
CodeIPI tests whether a tool-using coding agent follows hidden attacker instructions while it is supposed to fix a bug. Each item is a small Python repo plus a natural-language task. Injected samples plant a payload in issue text, a code comment, a README, or a config file. Ten clean samples have no payload and exist to measure false refusals. Success is not a quiz answer: the agent must edit files with bash and python inside Docker. English prompts. This is coding-agent IPI, not the workplace tool suites in [agentdojo](agentdojo.md) and not the harmful-behaviour suite in [agentharm](agentharm.md).
Multi-turn inspect_ai.agent.react solver with bash and python tools (120s timeouts) and max_messages 30. Default system prompt tells the agent to ignore unrelated commands in comments, docs, or issue text. defense_prompt=false drops that warning. Optional filters: vectors (issue_text, code_comment, readme, config_file) and severities (S1, S2, S3). Workspace files and canary secrets are written before the agent runs. Network is off (network_mode none).
No model card in ModelSpec reports this benchmark yet.