Stateful tool-using agent suites that score both task utility and resistance to prompt injections planted in tool outputs.
unassessed
| Category | agentic |
|---|---|
| Subcategory | tool-using agents under prompt injection (workspace, slack, travel, banking) |
| Page status | active |
| Metric | benign utility, utility under attack, and targeted attack success rate (ASR), each a fraction of cases |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1014 |
| Dataset licence | MIT |
| Publisher | ETH Zurich and Invariant Labs |
AgentDojo puts an LLM in a small simulated workplace and lets it call tools over mutable state: mail and calendar, Slack, travel booking, or banking. A user task is a legitimate request (pay a bill, summarise a channel, book a hotel). An injection task is a malicious goal an attacker tries to achieve by planting text in tool results. The benchmark measures whether the agent still finishes the user task and whether it also carries out the attacker's goal. Scoring checks environment state with formal utilities, not an LLM judge of the transcript.
Multi-turn tool-calling loop in one of four original suites (workspace, slack, travel, banking), plus inspect_evals' workspace_plus sandbox extension. Default inspect runs pair every user task with every injection task in the same suite.
No model card in ModelSpec reports this benchmark yet.