{
 "body": "\n## What it measures\n\nTheAgentCompany measures whether an LLM agent can act as a digital employee at a small software company: browsing internal web apps, writing and running code, and communicating with simulated coworkers to get real work done. Tasks are drawn from six roles - software engineering, project management, data science, HR, finance, and general administration - and range from short lookups to long, multi-step projects.\n\nThe paper frames this as a test of economically relevant automation, not narrow coding or QA skill: an agent must plan, use tools, and interact with other (simulated) people to succeed.\n\n## How it is scored\n\nEach task ships with an ordered checkpoint list plus a final pass/fail outcome, mixing deterministic and LLM-based evaluators. TheAgentCompany paper reports the strongest baseline agent (tested with both closed API and open-weight models) completing 30% of tasks fully autonomously, with partial checkpoint credit on many more. The Inspect Evals adaptation reuses this scoring but flags some checkpoint scorers as \"brittle\" (for example, string matching) and offers an \"improved\" scoring mode alongside the \"original\" one; the two are not interchangeable.\n\n## Dataset and licence\n\nThe original repository lists 175 tasks across the six roles above, distributed with an MIT licence, run inside Docker containers pre-populated with GitLab, the Plane project tracker, ownCloud, and RocketChat instances. The Inspect Evals integration is an adaptation of this repository and, as of this review, implements only the \"stage 1-2\" and test task subsets rather than the full 175.\n\n## Who publishes it\n\nCarnegie Mellon University researchers (Frank F. Xu, Yufan Song, Boxuan Li, and 18 further co-authors, including Graham Neubig) introduced TheAgentCompany in a paper submitted to arXiv in December 2024; a later revision was accepted to the NeurIPS 2025 Datasets and Benchmarks track. The authors maintain the code, environment, and a public leaderboard at the-agent-company.com.\n\n## Lineage\n\nTheAgentCompany is a standalone benchmark with no established predecessor. Inspect Evals maintains a partial reimplementation (`theagentcompany`) that should be treated as a distinct, smaller-coverage variant of the original 175-task suite rather than an identical restatement of it.\n\n## Saturation and contamination\n\nThe benchmark is not saturated: the paper reports the best agent solving only 30% of tasks autonomously, and harder, long-horizon tasks are described as \"still beyond the reach of current systems.\" Contamination risk is low for the interactive scoring itself, since success requires actually operating the live environment rather than reciting an answer, but the task instructions and environment setup are public on GitHub and could inform training or fine-tuning.\n\n## How to run it\n\nRun the original benchmark from github.com/TheAgentCompany/TheAgentCompany, which needs 30+ GB of disk space to host its Docker services, or run Inspect Evals task `theagentcompany`, noting it currently covers a subset of the full task list and offers both \"original\" and \"improved\" scoring modes. Record which task subset, scoring mode, and agent scaffold (for example OpenHands) were used, since these materially change the reported completion rate.\n\n## Reading the numbers\n\nA completion-rate score reflects how much of a simulated workday's tasks an agent can finish unsupervised in this specific environment; it does not establish safety, reliability, or general office competence, and results from the Inspect Evals subset are not directly comparable to the paper's full 175-task numbers. Look at checkpoint-level partial credit, not just full-task pass/fail, since many \"failures\" still made real progress.\n\nBecause the benchmark measures long-horizon, multi-tool tasks, small changes in the agent scaffold (browsing tools, memory, retry logic) can move scores substantially independent of the underlying model. Always report the task subset and scoring mode alongside the number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "theagentcompany",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "agentic",
  "contamination": {
   "note": "Tasks require live interaction with a self-hosted, stateful environment rather than a static answer key, so memorizing text does not solve a task; the environment and task instructions are nonetheless public on GitHub.",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 175,
   "size_note": "175 tasks across Software Development, Project Management, Data Science, Administrative, HR, and Finance roles, per the original repository; the Inspect Evals adaptation currently implements only the stage 1-2 and test task subsets.",
   "splits": "",
   "url": "https://github.com/TheAgentCompany/TheAgentCompany"
  },
  "freshness": {
   "luna-new-001 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "theagentcompany",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "theagentcompany",
  "last_updated": "",
  "leaderboard_url": "https://the-agent-company.com/#/leaderboard",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "TheAgentCompany evaluates whether an LLM agent can act as a digital worker in a small software company, browsing internal web apps, writing and running code, and messaging simulated coworkers to complete tasks drawn from software engineering, project management, data science, HR, finance, and admin roles.",
  "metric": {
   "baseline_note": "Each task is scored pass/fail plus partial credit via ordered checkpoints (0.0-1.0); the paper reports the best agent completing 30% of tasks autonomously.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "task success rate / checkpoint completion",
   "random_baseline": 0,
   "unit": "percent"
  },
  "name": "TheAgentCompany",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2412.14161",
   "title": "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks",
   "url": "https://arxiv.org/abs/2412.14161",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Frank F. Xu",
    "Yufan Song",
    "Boxuan Li",
    "Yuxuan Tang",
    "Kritanjali Jain",
    "Mengxue Bao",
    "Zora Z. Wang",
    "Xuhui Zhou",
    "Zhitong Guo",
    "Murong Cao",
    "Mingyang Yang",
    "Hao Yang Lu",
    "Amaad Martin",
    "Zhe Su",
    "Leander Maben",
    "Raj Mehta",
    "Wayne Chi",
    "Lawrence Jang",
    "Yiqing Xie",
    "Shuyan Zhou",
    "Graham Neubig"
   ],
   "org": "Carnegie Mellon University",
   "url": "https://github.com/TheAgentCompany/TheAgentCompany"
  },
  "released": "2024-12",
  "repo_url": "https://github.com/TheAgentCompany/TheAgentCompany",
  "saturation": {
   "as_of": "",
   "note": "The paper reports the most competitive agent completing 30% of tasks autonomously; harder long-horizon tasks remain largely unsolved, so scores still separate models.",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks (arXiv abstract)",
    "url": "https://arxiv.org/abs/2412.14161"
   },
   {
    "accessed": "2026-09-08",
    "title": "TheAgentCompany repository (README, LICENSE)",
    "url": "https://github.com/TheAgentCompany/TheAgentCompany"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inspect Evals TheAgentCompany integration",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/theagentcompany"
   },
   {
    "accessed": "2026-09-08",
    "title": "Inspect Evals repository",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals"
   }
  ],
  "status": "active",
  "subcategory": "computer-use agents",
  "summary": "TheAgentCompany has an agent complete 175 long-horizon professional tasks in a simulated software company to measure real-work automation.",
  "tags": [
   "agents",
   "tool-use",
   "computer-use",
   "long-horizon"
  ],
  "task_format": "The agent operates in a self-hosted environment (GitLab, the Plane project tracker, ownCloud, and RocketChat, all pre-populated with company data) and is given a natural-language task instruction; it must take actions over many steps to complete the task."
 }
}