{
 "body": "\n## What it measures\n\nTerminal-Bench 2.0 measures the same thing as the original Terminal-Bench: whether an AI agent can carry\nout a real task by issuing commands in a live, sandboxed terminal, across the same broad mix of domains\n(scientific workflows, infrastructure, data work, security and more). What changed from 1.0 is not the\nkind of task but how carefully each one was checked before release. The authors write plainly that they\n\"weren't satisfied with the level of verification\" in the original dataset \u2014 one example given is a task\nbuilt around scraping YouTube, which broke whenever YouTube's anti-bot defences changed \u2014 so every\nTerminal-Bench 2.0 task went through substantial manual and LM-assisted review.\n\n## How it is scored\n\nScoring is unchanged in kind from the original: an agent works in a sandboxed Docker terminal until it\nfinishes or times out, and an automated test script then checks whether the task was actually completed,\nwith results reported as a resolution rate plus an error bar reflecting variance across repeated runs.\nWhat changed is the infrastructure running that check: Terminal-Bench 2.0 shipped alongside Harbor, a\nrebuilt evaluation package supporting cloud-deployed containers at scale and rollout interfaces for\ntraining agents with RL or SFT, which replaced the original harness as the way the benchmark is actually\nrun.\n\n## Dataset and licence\n\nTerminal-Bench 2.0 shipped 89 tasks under an Apache-2.0 licence, hosted in a dedicated `terminal-bench-2`\nrepository separate from the original project's repository. A same-generation revision, Terminal-Bench\n2.1, later corrected 28 of those 89 tasks: nine had external dependencies that changed after the benchmark\nwas built, eight had resource budgets too tight for a valid solution to finish consistently, and the rest\nhad instructions that didn't match their tests. The 2.1 release states that after these fixes, no task in\nthe set is left completely unsolved by every agent evaluated.\n\n## Who publishes it\n\nTerminal-Bench 2.0 and Harbor were announced by Mike Merrill and Alex Shaw for the Terminal-Bench Team;\nTerminal-Bench 2.1 credits Kelly Buchanan as lead. Neither announcement carries an explicit publish date\non the page itself; the Internet Archive's earliest capture of the 2.0 announcement is 2025-11-11, which\nthis page uses as an approximate release date. The project is hosted under the Harbor Framework\norganisation on GitHub.\n\n## Lineage\n\nTerminal-Bench 2.0 is the direct successor to the original Terminal-Bench (`terminal_bench`), rebuilt for\nquality rather than task variety. Terminal-Bench 2.1 is a revision within the same generation (same 89\ntasks, 28 corrected), not a new major version, and this page treats the two together. The project moved on\nto further major releases \u2014 v3.0.0 (July 2026) and v4.0.0 (August 2026) \u2014 hosted in the main\n`terminal-bench` repository and apparently merged with a separate, harder task set (\"frontier-bench\" in\nthat repository's own release history); those later versions are not catalogued here.\n\n## Saturation and contamination\n\nOn the live Terminal-Bench 2.1 leaderboard, the top entry as of 2026-09-08 is GPT-6 Astra (high reasoning\neffort) paired with the Codex agent at 87.4% (\u00b11.8%), dated 2026-09-03, just ahead of a second entry at\n83.8%. This is 2.1-revised data, not raw 2.0: the 2.1 release's own before/after comparison shows most\nagent-model pairs scoring measurably higher on 2.1 than on 2.0 under an identical setup \u2014 Claude Opus 4.6\nwith Claude Code, for instance, rose from 58.0% to 70.1% \u2014 so the two are not directly comparable. The\nleaderboard mixes vendor-native coding CLIs (Codex, Claude Code, Cursor CLI, Gemini CLI) with the\nproject's own neutral Terminus 2 agent, and rows on the same model score differently by agent.\nContamination risk sits at medium: tasks are hand-built and heavily reviewed rather than mined from a\npublic corpus, but 2.1's own release notes describe outside users, including a named vendor, finding and\nreporting problems with specific 2.0 tasks, so task content is discussed publicly and could reach later\ntraining data.\n\n## How to run it\n\nTerminal-Bench 2.0 and 2.1 run on Harbor rather than the original harness, invoked as `harbor run -d\nterminal-bench/terminal-bench-2-1/...`. Terminus 2, an updated version of the project's deliberately\nminimal reference agent, is one of several agents on the leaderboard; scores from Terminus 2 and from a\nfull-featured vendor CLI on the same underlying model are not the same measurement, since the latter\nbrings its own tools, prompting and retry logic.\n\n## Reading the numbers\n\nA Terminal-Bench 2.x score is a stronger signal than a 1.0 score for the same nominal resolution rate,\nbecause more of its tasks have been checked to actually be solvable and correctly graded. It still\ndescribes agentic, tool-using competence across a broad task mix rather than any one domain. Given how much\nthe 2.0-to-2.1 fix moved scores for identical agent-model pairs, and how much the field has moved since\nNovember 2025, a \"Terminal-Bench 2\" number should be read together with its exact revision (2.0 vs 2.1) and\nits agent scaffold before being compared to another source's number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "terminal_bench_2",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 82.0,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5",
   "model_id": "openai/gpt-5",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5 Chat (latest)",
   "model_id": "openai/gpt-5-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Chat (latest)",
   "model_id": "openai/gpt-5-3-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex",
   "model_id": "openai/gpt-5-3-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.3 Codex Spark",
   "model_id": "openai/gpt-5-3-codex-spark",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 75.1,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.5,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 65.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 65.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2",
   "model_id": "openai/gpt-5-2",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 64.0,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Chat",
   "model_id": "openai/gpt-5-2-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 64.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.2 Codex",
   "model_id": "openai/gpt-5-2-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 64.0,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GLM 5.1",
   "model_id": "zhipu/glm-5-1",
   "provider": "zhipu",
   "provider_display": "Z.ai (Zhipu AI)",
   "score": 63.5,
   "source": "zai-org-model-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5",
   "model_id": "anthropic/claude-opus-4-5-20251101",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 59.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.5 (latest)",
   "model_id": "anthropic/claude-opus-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 59.3,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.6",
   "model_id": "anthropic/claude-sonnet-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 59.1,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Muse Spark",
   "model_id": "meta/muse-spark",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 59.0,
   "source": "meta-blog, officechai, artificial-analysis"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1",
   "model_id": "openai/gpt-5-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 52.8,
   "source": "lmarena.ai, provider-reports, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Chat",
   "model_id": "openai/gpt-5-1-chat-latest",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 52.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex",
   "model_id": "openai/gpt-5-1-codex",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 52.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex Max",
   "model_id": "openai/gpt-5-1-codex-max",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 52.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.1 Codex mini",
   "model_id": "openai/gpt-5-1-codex-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 52.8,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 46.4,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 46.4,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 46.4,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 46.4,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3-Coder 480B-A35B Instruct",
   "model_id": "qwen/qwen3-coder-480b-a35b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 37.5,
   "source": "lmarena.ai, provider-reports"
  }
 ],
 "page": {
  "aliases": [
   "Terminal-Bench 2",
   "TB2",
   "Terminal-Bench 2.1"
  ],
  "category": "agentic",
  "contamination": {
   "note": "Tasks are hand-crafted and heavily manually verified rather than mined from a pre-existing public corpus, which limits initial exposure. But 2.1's own release notes describe outside users (including a named vendor) finding and reporting problems with specific 2.0 tasks, so task content and behaviour are actively discussed publicly; once a task and its solution are public, later-trained models could plausibly encounter them.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 89,
   "size_note": "Terminal-Bench 2.0 shipped 89 tasks, substantially re-verified versus the original. Terminal-Bench 2.1, a same-generation revision, later fixed 28 of those 89 tasks \u2014 9 broken by changed external dependencies, 8 with resource budgets too tight for a valid solution to finish reliably, and the rest from instructions that did not match their tests \u2014 and reports that after the fix, no task in the set is left completely unsolved by every evaluated agent.\n",
   "splits": "Terminal-Bench 2.0 (89 tasks) and its 2.1 revision (same 89, 28 corrected)",
   "url": "https://github.com/harbor-framework/terminal-bench-2"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice M",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Runs on Harbor (`harbor run -d terminal-bench/terminal-bench-2-1/...`), which replaced the original Terminal-Bench harness as of this release. Terminus 2, an updated version of the project's minimal reference agent, is one of several agents scored on the leaderboard alongside vendor-native CLIs (Codex, Claude Code, Cursor CLI, Gemini CLI); which one is used materially affects both resolution rate and the cost/token figures shown on the same board.\n"
  },
  "id": "terminal_bench_2",
  "last_updated": "",
  "leaderboard_url": "https://www.tbench.ai/leaderboard/terminal-bench/2.1",
  "lineage": {
   "family": "",
   "predecessor": "terminal_bench",
   "successors": [],
   "variants": []
  },
  "measures": "Terminal-Bench 2.0 measures the same thing as the original Terminal-Bench: whether an AI agent can carry out a real task by issuing commands in a live, sandboxed terminal. The task domains and text-only interface are unchanged; what changed is quality control. The authors write that they \"weren't satisfied with the level of verification\" in the original dataset \u2014 for example, a task that scraped YouTube broke whenever YouTube's anti-bot defences changed \u2014 so 2.0 puts each task through substantial manual and LM-assisted review before inclusion.\n",
  "metric": {
   "baseline_note": "No formal human baseline published; scores are reported with error bars reflecting variance across repeated runs.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "task resolution rate (tasks passed / tasks attempted)",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "Terminal-Bench 2.0",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Mike Merrill",
    "Alex Shaw"
   ],
   "org": "The Terminal-Bench Team, hosted under the Harbor Framework project",
   "url": "https://www.tbench.ai/"
  },
  "released": "2025-11",
  "repo_url": "https://github.com/harbor-framework/terminal-bench-2",
  "saturation": {
   "as_of": "2026-09",
   "note": "On the live Terminal-Bench 2.1 leaderboard (accessed 2026-09-08), the top entry is GPT-6 Astra (high reasoning effort) with the Codex agent at 87.4% (\u00b11.8%), dated 2026-09-03, ahead of a second entry at 83.8%. This is 2.1-revised data, not raw 2.0: the 2.1 release's own before/after comparison shows most agent-model pairs scoring measurably higher on 2.1 than on 2.0 under an identical setup (for example Claude Opus 4.6 with Claude Code rose from 58.0% on 2.0 to 70.1% on 2.1), so a 2.0 score and a 2.1 score are not directly interchangeable. The top rows also mix a vendor-native coding CLI (Codex) with the project's own neutral Terminus 2 agent lower on the same board, which further affects comparability.\n",
   "status": "watch",
   "top_score": 87.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Terminal-Bench 2.0 and Harbor",
    "url": "https://www.tbench.ai/news/announcement-2-0"
   },
   {
    "accessed": "2026-09-08",
    "title": "Terminal-Bench 2.1",
    "url": "https://www.tbench.ai/news/terminal-bench-2-1"
   },
   {
    "accessed": "2026-09-08",
    "title": "Terminal-Bench 2.1 leaderboard",
    "url": "https://www.tbench.ai/leaderboard/terminal-bench/2.1"
   },
   {
    "accessed": "2026-09-08",
    "title": "Terminal-Bench 2.0 and Harbor (Wayback Machine capture, 11 Nov 2025)",
    "url": "http://web.archive.org/web/20251111104155/https://www.tbench.ai/news/announcement-2-0"
   }
  ],
  "status": "active",
  "subcategory": "general-purpose terminal / command-line agent task completion, verified task set",
  "summary": "A harder, more heavily verified 89-task remake of Terminal-Bench, released with the Harbor evaluation package; Terminal-Bench 2.1 later patched 28 of its tasks.",
  "tags": [
   "agentic",
   "terminal",
   "command-line",
   "docker",
   "tool-use",
   "verified"
  ],
  "task_format": "Unchanged from the original Terminal-Bench: an agent receives a task instruction and a sandboxed Docker terminal, and must complete the task through shell commands. Terminal-Bench 2.0 shipped alongside Harbor, a rebuilt evaluation package (cloud-deployed containers, rollout interfaces for RL/SFT training, a simpler any-agent interface) that replaced the original harness as the way tasks are run and graded.\n"
 }
}