214 realistic web tasks such as monitoring listings or comparing prices, scored by an automatic answer-matching function; no system has cleared half credit on the public leaderboard.
unassessed
| Category | agentic |
|---|---|
| Subcategory | realistic, time-consuming web tasks scored by automatic answer matching |
| Page status | active |
| Metric | accuracy (custom partial-credit scorer) and answer rate |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 214 |
| Dataset licence | Apache-2.0, per the Hugging Face dataset card |
| Publisher | Tel Aviv University, with Allen Institute for AI, University of Pennsylvania, University of Washington and Princeton University |
AssistantBench gives a model or agent a natural-language request that a person might actually want handled on the open web -- monitoring a real-estate listing, comparing ticket prices across dates, locating a nearby business that meets several criteria -- and checks whether it can produce the correct answer. The authors built it specifically because prior web-agent benchmarks tended to use short, artificial tasks; AssistantBench's 214 tasks are instead designed to be realistic and, in many cases, genuinely time-consuming for a human to complete by hand, spanning many sites and domains rather than one sandboxed environment. Answers take several forms -- a number, a short string, a list, or a small JSON object -- rather than always being a single fact, so scoring has to handle partial credit rather than plain exact match.
A natural-language task description in, a free-form answer out. The reference implementation (inspect_evals) runs five variants of the same 214-task set: closed-book zero-shot, closed-book one-shot, web-search zero-shot, web-search one-shot, and a web-browser-plus-search agent variant the authors did not run themselves. The original paper also evaluated a SeeAct web-navigation agent and the authors' own SeePlanAct (SPA) agent, neither of which inspect_evals reproduces, since both need scaffolding the framework does not support.
No model card in ModelSpec reports this benchmark yet.