{
 "body": "\n## What it measures\n\nOSWorld measures whether a multimodal agent can operate a real desktop the way a person would.\nGiven a natural-language instruction and a live view of a virtual machine, usually a screenshot and\nsometimes an accessibility tree, the agent must decide which application to open, what to click, type\nor drag, and when the task is finished. Tasks run against real software: office suites, browsers, file\nmanagers and other GUI utilities, and many span OS file I/O and multiple applications rather than a\nsingle isolated action. The modality is image plus text; the primary environment is Ubuntu, with\nWindows and macOS support added since the original release.\n\n## How it is scored\n\nEach of the 369 tasks ships with an initial-state setup config and a task-specific, execution-based\nchecker script that inspects the resulting file system, application state or output after the agent\nfinishes. Success is binary per task, and the headline metric is success rate: successful tasks divided\nby tasks attempted, reported as a percentage. The original paper measured a human baseline of 72.36% by\nhaving people perform the same 369 tasks, and reported that the strongest agent tested at publication\nreached only 12.24%, struggling mainly with GUI grounding (clicking the right element) and operational\nknowledge (knowing an application's workflow). Eight of the 369 tasks require a Google Drive account and\nare sometimes excluded, leaving a commonly reported 361-task variant.\n\n## Dataset and licence\n\n369 tasks (361 excluding the Google-Drive-dependent ones), each built from a real-world computer-use\ncase with a config describing initial state, instruction and a task-specific evaluation script. Tasks\nare grouped into categories such as Office, Daily and Professional workflows. The GitHub repository is\nApache-2.0 licensed per its LICENSE file; the project's companion website separately carries a Creative\nCommons Attribution-ShareAlike 4.0 notice for its own page content. There is no single held-out set of\ntextual answers, since grading runs an executable checker against the final machine state, and the task\nconfigs and checkers are themselves public in the repository.\n\n## Who publishes it\n\nOSWorld comes from a multi-institution academic collaboration: authors from the University of Hong Kong,\nSalesforce Research, Carnegie Mellon University and the University of Waterloo, led by Tianbao Xie with\n16 co-authors including Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong and Tao Yu. The paper,\n\"OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments,\" appeared\non arXiv in April 2024 and was accepted at NeurIPS 2024. The xlang-ai group continues to maintain the\nGitHub repository and project site, which hosts a verified leaderboard that researchers submit\nimplementations to for official scoring.\n\n## Lineage\n\nOSWorld names no direct predecessor; the paper frames it as the first real, scalable, cross-OS computer\nenvironment for agent evaluation, rather than a scripted or web-only sandbox. It has since spawned\nfollow-on projects from the same group and wider community, including OSWorld-Verified (task-config\nfixes and AWS support for parallel evaluation) and OSWorld-MCP (testing whether computer-use agents can\ninvoke Model Context Protocol tools). Neither has its own id or page in this repository yet.\n\n## Saturation and contamination\n\nAt publication (April 2024) the gap between the human baseline (72.36%) and the best evaluated agent\n(12.24%) was large, and the paper treats the benchmark as far from solved at that point \u2014 open, as of\nApril 2024. A more recent, dated leaderboard reading beyond the original paper's numbers was not\nestablished from the sources reviewed here, so treat \"open\" as reflecting the 2024 publication, not\nnecessarily today's state of the field. Because task configs, checker scripts and reference solutions\nlive in the open GitHub repository since 2024, a model's pretraining data could plausibly include them;\nthe authors do not describe a specific contamination mitigation, though execution-based grading against\na live environment makes rote memorization of a \"correct answer\" less directly useful than for a\nquestion-answering benchmark, since the agent still has to act correctly.\n\n## How to run it\n\nThe reference implementation is at github.com/xlang-ai/OSWorld, which provisions virtual machines\n(VMware, VirtualBox, Docker, Modal, Daytona or AWS for Ubuntu; Docker for Windows) and drives them\nthrough an agent loop that issues actions until the agent signals completion or hits a step limit.\nComparability depends on the observation space (screenshot only versus screenshot plus accessibility\ntree), the step budget, and which task subset (369 versus 361) was used. None of these were confirmed as\nstandardized entries in lm-evaluation-harness, inspect_evals, HELM, OpenCompass or BIG-bench's task lists.\n\n## Reading the numbers\n\nA high OSWorld score means an agent can chain together multi-step, multi-application desktop work, the\nkind a human assistant would do, without being told the exact clicks. It does not say how the agent\nperforms on tasks outside the 369-task distribution, on operating systems or applications not covered,\nor under UI changes the checker scripts do not anticipate. Because grading is execution-based, a passing\ntask genuinely happened in a live environment, a stronger signal than a model merely claiming success in\ntext. Before comparing two reported scores, check whether they used the full 369-task set or the\n361-task variant, and what observation space the agent had.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "osworld",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 79.6,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 75.0,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 72.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  }
 ],
 "page": {
  "aliases": [
   "OS-World"
  ],
  "category": "agentic",
  "contamination": {
   "note": "Task configs, initial-state setup and checker scripts are public in the GitHub repository since April 2024, so a model could have seen them in pretraining. Execution-based grading against a live environment limits how useful verbatim memorization is compared with a text-answer benchmark, but the authors do not describe a specific mitigation.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0 (repository licence, from the GitHub LICENSE file; the project website separately displays a CC BY-SA 4.0 notice for its own page content)",
   "modalities": [
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 369,
   "size_note": "369 tasks across real desktop and web applications; 8 require a Google Drive account and are sometimes excluded, leaving 361.",
   "splits": "single evaluation set; no train/test split described in the sources reviewed",
   "url": "https://github.com/xlang-ai/OSWorld"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice H",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "osworld",
  "last_updated": "",
  "leaderboard_url": "https://os-world.github.io/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "OSWorld gives an agent a natural-language instruction and a live view of a virtual machine, usually a screenshot and sometimes an accessibility tree, and asks it to operate real desktop and web applications to complete the task: finding files, editing documents, configuring settings, and other workflows that span multiple applications and OS file I/O rather than a single scripted action.",
  "metric": {
   "baseline_note": "Human baseline of 72.36% measured by the paper's authors performing the same 369 tasks; the strongest agent evaluated in the original paper reached 12.24%.",
   "direction": "higher_is_better",
   "human_baseline": 72.36,
   "max_score": 100.0,
   "name": "success rate",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "OSWorld",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2404.07972",
   "title": "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments",
   "url": "https://arxiv.org/abs/2404.07972",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Tianbao Xie",
    "Danyang Zhang",
    "Jixuan Chen",
    "Xiaochuan Li",
    "Siheng Zhao",
    "Ruisheng Cao",
    "Toh Jing Hua",
    "Zhoujun Cheng",
    "Dongchan Shin",
    "Fangyu Lei",
    "Yitao Liu",
    "Yiheng Xu",
    "Shuyan Zhou",
    "Silvio Savarese",
    "Caiming Xiong",
    "Victor Zhong",
    "Tao Yu"
   ],
   "org": "University of Hong Kong, Salesforce Research, Carnegie Mellon University, University of Waterloo",
   "url": "https://os-world.github.io/"
  },
  "released": "2024-04",
  "repo_url": "https://github.com/xlang-ai/OSWorld",
  "saturation": {
   "as_of": "2024-04",
   "note": "Best agent result from the original paper; a more recent dated leaderboard reading was not established from the sources reviewed. The human baseline (72.36%) sits far above this, so the gap may have closed since publication.",
   "status": "open",
   "top_score": 12.24
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments (arXiv)",
    "url": "https://arxiv.org/abs/2404.07972"
   },
   {
    "accessed": "2026-09-07",
    "title": "xlang-ai/OSWorld GitHub repository",
    "url": "https://github.com/xlang-ai/OSWorld"
   },
   {
    "accessed": "2026-09-07",
    "title": "OSWorld project site (canonical URL os-world.github.io redirects here)",
    "url": "http://osworld-v1.xlang.ai/"
   }
  ],
  "status": "active",
  "subcategory": "computer-use / GUI agents",
  "summary": "Tests whether a multimodal agent can complete open-ended tasks in a real, live desktop operating system.",
  "tags": [
   "gui-agents",
   "computer-use",
   "multimodal-agents",
   "desktop-automation"
  ],
  "task_format": "screenshot (optionally plus accessibility tree) as observation; agent issues GUI actions (click, type, drag, hotkey) in a loop until it signals completion or hits a step limit"
 }
}