MCP-Bench scores agents on 104 fuzzy multi-step tasks that must call tools across 28 live MCP servers (250 tools) without being told the tool names.
unassessed
| Category | agentic |
|---|---|
| Subcategory | tool-using agents on live Model Context Protocol servers |
| Page status | active |
| Metric | overall score (rule-based schema checks plus LLM-judge axes) |
| Direction | higher_is_better |
| Dataset size | 104 |
| Publisher | Accenture Center for Advanced AI, with UC Berkeley |
The agent gets a vague English instruction, a pool of MCP servers, and must finish a real-world style job: search papers, plan travel, convert units, query weather, and similar. Servers expose complementary tools, so a task often needs a chain, not one call. Instructions are fuzzy: they do not name the tool. Extra distraction servers are in the pool by default. The run mixes single-server tasks with two-server and three-server combinations. Scoring splits rule-based schema checks from an o4-mini judge of completion, tool use, and planning.
Multi-round tool calling over live MCP (default max 20 rounds). Official task files: 56 single-server, 30 two-server, 18 three-server (104 total). Fuzzy descriptions on; concrete text and dependency notes are held for the judge. Time MCP is a resident server.
No model card in ModelSpec reports this benchmark yet.