Tests LLMs as interactive agents across eight environments (OS, database, knowledge graph, card game, puzzles, household, web shopping, web browsing); most modern harnesses implement only its OS slice.
unassessed
| Category | agentic |
|---|---|
| Subcategory | multi-environment interactive LLM-as-agent evaluation (8 environments) |
| Page status | active |
| Metric | environment-specific metric (Success Rate, F1, Reward, or Game Progress), combined into an overall weighted score across all eight environments |
| Direction | higher_is_better |
| Dataset size | 1360 |
| Dataset licence | Apache-2.0 |
| Publisher | Tsinghua University, with The Ohio State University |
AgentBench puts a model in the loop of a multi-turn interaction with a real or simulated environment and asks it to complete a task by issuing actions -- shell commands, SQL queries, dialogue moves, game actions -- rather than answering a single question. It spans eight distinct environments, five built specifically for the benchmark (Operating System, Database, Knowledge Graph, a Digital Card Game, and Lateral Thinking Puzzles) and three adapted from existing published environments (ALFWorld for household tasks, WebShop for web shopping, and Mind2Web for web browsing). Each environment exercises a different mix of instruction-following, long-horizon planning and environment-grounded decision-making.
Multi-turn interaction loop, up to 5-35 estimated turns depending on environment; the model issues an action each turn (bash/SQL command, dialogue act, game move) and receives an environment observation in response, until it completes the task, fails, or hits a turn limit.
No model card in ModelSpec reports this benchmark yet.