Terminal-Bench

Terminal-Bench 1.0 measures whether an AI agent can complete real command-line tasks in a sandboxed Docker terminal, graded by automated tests; superseded by later major versions.

Also known as: Terminal-Bench 1.0, Terminal-Bench-Core, T-Bench

unassessed

What it measures

Terminal-Bench gives an agent a natural-language instruction and a live, sandboxed terminal (a Docker container, sometimes several linked containers) and asks it to complete the task by issuing commands, the way a person would work at a shell. Tasks at release covered scientific workflows, network configuration, games, data analysis, calling APIs and fixing security vulnerabilities, deliberately going beyond software-engineering-only benchmarks like SWE-bench. Everything happens in text, matching how language models are trained, but the range of tools, state and multi-step planning required is broader than a single coding task.

Task format

An agent receives a task instruction and a terminal session inside a Docker environment. The Terminal-Bench harness orchestrates the agent (installed directly in the container, integrated via a Python interface, or connected through an MCP server exposing a tmux session), logs its actions, and checks final container state against a task-specific automated test script after the agent finishes or times out. Each task also ships a human-verified reference solution.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub