A GuardAgent safety eval: generate and execute code that grants or denies a SeeAct web action under six user-constraint rules.
unassessed
| Category | safety |
|---|---|
| Subcategory | web-agent guardrails (SeeAct + user constraints) |
| Page status | active |
| Metric | accuracy (comprehensive correctness: decision plus violation on DENY) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 200 |
| Dataset licence | inspect_evals MIT; GuardAgent paper HTML states CC BY 4.0; original data also hosted at guardagent/dataset |
| Publisher | University of Georgia, University of Chicago, UIUC, UT Austin, UC Berkeley, Emory, Virtue AI (GuardAgent authors) |
Mind2Web-SC tests a guard model, not a web navigator. The model sees a user task from Mind2Web, a synthetic user profile (age plus boolean flags for membership, vaccine, driver's licence, and domestic status), and a SeeAct agent output. It must decompose the applicable safety rule, write Python guardrail code, and decide GRANT or DENY. On DENY it must also name the violated rule. English text plus executed code. It is not the [Mind2Web](mind2web.md) action-prediction task.
Few-shot code generation (inspect default num_shots=3) inside a Docker sandbox. inspect_evals asks for task decomposition and guardrail code, runs the code, then parses "Action granted" / "Action denied" and an optional "Violation:" line. Paper metrics include label precision/recall and comprehensive control accuracy; inspect reports accuracy() plus accuracy grouped by domain.
No model card in ModelSpec reports this benchmark yet.