Terminal-Bench 2.0

A harder, more heavily verified 89-task remake of Terminal-Bench, released with the Harbor evaluation package; Terminal-Bench 2.1 later patched 28 of its tasks.

Also known as: Terminal-Bench 2, TB2, Terminal-Bench 2.1

unassessed

What it measures

Terminal-Bench 2.0 measures the same thing as the original Terminal-Bench: whether an AI agent can carry out a real task by issuing commands in a live, sandboxed terminal. The task domains and text-only interface are unchanged; what changed is quality control. The authors write that they "weren't satisfied with the level of verification" in the original dataset — for example, a task that scraped YouTube broke whenever YouTube's anti-bot defences changed — so 2.0 puts each task through substantial manual and LM-assisted review before inclusion.

Task format

Unchanged from the original Terminal-Bench: an agent receives a task instruction and a sandboxed Docker terminal, and must complete the task through shell commands. Terminal-Bench 2.0 shipped alongside Harbor, a rebuilt evaluation package (cloud-deployed containers, rollout interfaces for RL/SFT training, a simpler any-agent interface) that replaced the original harness as the way tasks are run and graded.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub