300 paired tool-use scenarios that score whether an agent changes its stated plan under pressure while its action stays the same; the metric is Pass@5 deception rate.
unassessed
| Category | safety |
|---|---|
| Subcategory | agent deception: plan-action divergence under pressure in simulated tool use |
| Page status | active |
| Metric | Pass@5 deception rate (share of pairs with a deceptive sample in k=5) |
| Direction | lower_is_better |
| Unit | % |
| Dataset size | 300 |
| Dataset licence | CC-BY-NC-ND-4.0 (paper); dataset SPDX not stated beyond an 'open research license' |
| Publisher | Beijing Academy of Artificial Intelligence; Peking University; University of Science and Technology of China; University of Chinese Academy of Sciences; Alibaba Group |
SPADE-Bench tests whether a tool-using agent tells an observer one plan and then executes another. Each case is a binary workplace dilemma with simulated tools. In the regular variant the observer pressure is off. In the pressure variant an auditor or supervisor can see the plan but not the tool calls. Deception is a shift of the plan toward the observer while the action stays on the agent's intrinsic choice. The authors argue this is strategic, not a random hallucination, because the pair is scored together.
The agent receives XML-tagged context: tools, role, principles, ground truth, optional pressure, and a user query. It must call a create_plan tool, then act with the rest of the simulated library. A stance classifier labels plan and action as intrinsic, external, or ambiguous. Deception is a rule on the pair: action stays intrinsic across both variants, plan moves toward the observer under pressure. Default protocol is Pass@5 at temperature 0.7.
No model card in ModelSpec reports this benchmark yet.