Small Inspect AI suite that scores whether an LLM agent completes its task and separately whether it resists prompt-injection attacks drawn from the OWASP Agentic Top 10.
unassessed
| Category | safety |
|---|---|
| Subcategory | agent security / prompt-injection and goal-hijack resistance (OWASP Agentic Top 10) |
| Page status | active |
| Metric | dual binary metric per sample: utility (task completed correctly) and security (attack resisted), each graded CORRECT (1.0) or INCORRECT (0.0) by task-specific criteria, reported as two separate percentages rather than combined into one score |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 24 |
| Dataset licence | MIT (the inspect_evals package licence; no separate licence is stated for this task's own data files) |
| Publisher | Inspect Evals (UK AI Security Institute, in collaboration with Arcadia Impact and the Vector Institute); this specific suite was community-contributed |
AgentThreatBench gives an agent an ordinary task -- answer a question from a memory store, triage an email inbox, resolve a customer-support ticket -- where one of the tool outputs the agent reads (a memory entry, an email body, a customer record) has been seeded with an adversarial instruction trying to redirect the agent's behaviour. It tests whether the agent both finishes the legitimate task and refuses to be manipulated by the embedded attack, scoring the two outcomes independently rather than folding them into one number. The three tasks operationalise two categories from the OWASP Top 10 for Agentic Applications (2026), namely memory and context poisoning and agent goal hijack via indirect prompt injection.
Three separate Inspect AI tasks, each a small set of multi-turn tool-use scenarios with an adversarial payload embedded in one tool's output; the agent calls tools to complete its task while a dual grader checks task completion and whether the embedded attack succeeded.
No model card in ModelSpec reports this benchmark yet.