{
 "body": "\n## What it measures\n\nScreenSpot-Pro tests GUI grounding \u2014 locating the exact on-screen element an instruction refers to \u2014\nin professional software rather than the everyday consumer apps earlier grounding benchmarks focused\non. Given a screenshot and a natural-language instruction, a model must output the location of the\ntarget element. Every image is an authentic, expert-captured screenshot from real professional\nworkflows (CAD, scientific computing, creative tools, IDEs and more) at native high resolution, so\ntargets are far smaller relative to the frame than on earlier benchmarks: about 0.07% of the\nscreenshot area on average, against 2.01% on the original ScreenSpot benchmark this one extends.\n\nThe authors built it because professional software had been left out of prior GUI agent research,\nwhich concentrated on web browsing and mobile use, and because high resolution, small targets and\ncluttered interfaces degrade grounding accuracy sharply compared to simpler consumer screens.\n\n## How it is scored\n\nA prediction is correct when the model's output point falls inside the human-annotated bounding box\nfor the target element, the click-in-box protocol used by the original ScreenSpot benchmark this one\nextends. Overall accuracy is a micro-average across the benchmark's 1,581 instructions, with a\nper-category breakdown across the six application groupings. No random baseline is meaningful given\nhow small targets are; the paper instead reports GPT-4o, used zero-shot as a general-purpose model\nrather than a specialized grounder, scoring only 0.9% \u2014 illustrating how much this task rewards\npurpose-built grounding models over general chat models.\n\n## Dataset and licence\n\nThe benchmark comprises 1,581 instructions over screenshots spanning 23 applications across five\nindustry groupings (development and programming, creative, CAD and engineering, scientific and\nanalytical, office) plus operating-system-level tasks, across Windows, macOS and Linux. Experts with\nat least five years of experience in each application recorded the tasks themselves, using a\npurpose-built capture tool triggered by a shortcut key so annotations reflected real work rather than\nstaged scenarios; each instance was reviewed by at least two annotators. About 62.6% of targets are\ntext elements, the remainder icons. The dataset and code are released under the MIT licence and\ndistributed through Hugging Face (`likaixin/ScreenSpot-Pro`) and GitHub.\n\n## Who publishes it\n\nThe paper was written by Kaixin Li, Ziyang Meng, Hongzhan Lin, Ziyang Luo, Yuchen Tian, Jing Ma,\nZhiyong Huang and Tat-Seng Chua. The project and dataset were first released in January 2025, with the\narXiv preprint posted that April and the work later presented at a Workshop on Reasoning and Planning\nfor Large Language Models. The authors maintain the dataset, evaluation code and a public leaderboard\ndirectly; the benchmark has since been cited or adopted as an evaluation set by several other GUI-agent\nprojects, including Microsoft's OmniParser v2, Qwen2.5-VL and UI-TARS.\n\n## Lineage\n\nScreenSpot-Pro is a direct successor to ScreenSpot (Cheng et al.), extending it from cropped,\neasy-usage screenshots to authentic high-resolution professional software, and is commonly cited\nalongside ScreenSpot-v2 as part of the same evaluation lineage; neither has its own page in this\nrepository yet. This repository tracks one direct variant, `screenspot_pro_tools`, covering scores\nproduced when a model uses an iterative search or agentic loop rather than single-shot grounding.\n\n## Saturation and contamination\n\nThe benchmark opened very hard: the best existing grounding model at release, OS-Atlas-7B, scored\n18.9%, and the paper's own proposed method, ScreenSeekeR (a cascaded visual search guided by a\nplanner model), pushed that to 48.1% without any additional training. The project's own leaderboard,\nfetched directly for this page, currently lists its top entry at 82.7%, held by a specialized\ngrounding model using an iterative \"zoom-in\" strategy rather than a general frontier chat model \u2014\nconsistent with the paper's own finding that reducing the search area is the single most effective\nlever on this task. Scores still span a wide range across the leaderboard, so the benchmark separates\nmodels well and is not saturated. Contamination risk is comparatively low, since the images are\nauthentic screen captures rather than text scraped from the public web.\n\n## How to run it\n\nThe authors' repository ships `eval_screenspot_pro.py` (and a parallel variant) against an\nOpenAI-compatible API; results are typically submitted to the project's own leaderboard rather than\nrun through a third-party harness such as lm-evaluation-harness or inspect_evals, neither confirmed to\ninclude this task. Because top leaderboard entries mix single-shot grounding, iterative zoom or crop\nstrategies, and fully agentic planner-guided search, comparing two scores requires checking which\nstrategy each used \u2014 the distinction `screenspot_pro_tools` is meant to isolate.\n\n## Reading the numbers\n\nA high ScreenSpot-Pro score indicates a model, or a model paired with a grounding strategy, can find\nsmall, specific UI elements in dense professional software \u2014 a prerequisite for reliable computer-use\nagents in real work tools, not just consumer apps. It does not measure whether an agent can complete a\nmulti-step task in that software, only whether it can locate one element given a clear instruction.\nBecause the leaderboard is dominated by specialized fine-tuned grounders rather than general frontier\nchat models, a strong general-purpose model's plain score can understate its usefulness once paired\nwith a search or zoom strategy \u2014 why the gap between this page and `screenspot_pro_tools` often\nmatters more than either number alone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "screenspot_pro",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 79.5,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 57.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  }
 ],
 "page": {
  "aliases": [
   "ScreenSpot Pro"
  ],
  "category": "agentic",
  "contamination": {
   "note": "Screenshots are authentic captures of real professional software use rather than text drawn from the open web, so the usual internet-scale pretraining leakage pathway is less direct. The paper does not discuss contamination as a concern; risk mainly comes from the dataset and its answer bounding boxes being publicly downloadable, which could let later models be tuned specifically on these images.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "image",
    "text"
   ],
   "public_test_set": true,
   "size": 1581,
   "size_note": "1,581 instructions over authentic screenshots spanning 23 applications across five industries (development and programming, creative, CAD and engineering, scientific and analytical, office) plus a sixth \"operating system commons\" category, across three operating systems (Windows, macOS, Linux). About 62.6% of targets are text elements and the remainder are icons.\n",
   "splits": "single evaluation set, no train/test split",
   "url": "https://huggingface.co/datasets/likaixin/ScreenSpot-Pro"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice L",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The authors' own repository ships eval_screenspot_pro.py and a parallel variant, requiring an OpenAI-compatible API key; results are typically submitted to the project's own leaderboard rather than run through a third-party harness.\n"
  },
  "id": "screenspot_pro",
  "last_updated": "2026-08",
  "leaderboard_url": "https://gui-agent.github.io/grounding-leaderboard/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "screenspot_pro_tools"
   ]
  },
  "measures": "ScreenSpot-Pro tests GUI grounding in professional software: given a natural-language instruction and a screenshot, a model must locate the precise on-screen element the instruction refers to. Unlike earlier grounding benchmarks built around everyday consumer apps and mobile screens, every image here is an authentic, expert-captured screenshot from real professional workflows in fields like CAD, scientific computing, creative software and IDEs, at native high resolution (over 1080p) rather than a cropped or downscaled view. Targets are correspondingly tiny: on average a target occupies about 0.07% of the screenshot area, versus 2.01% on the original ScreenSpot benchmark this one extends.\n",
  "metric": {
   "baseline_note": "A prediction counts as correct when the model's output point falls inside the human-annotated bounding box for the instruction's target element, the same click-in-box protocol used by the original ScreenSpot benchmark this one extends; overall accuracy is the share of the 1,581 instructions solved this way (the project reports a micro-average across categories). No formal random or human baseline is published; the paper instead reports that GPT-4o, a general frontier model used zero-shot, scored only 0.9% direct grounding accuracy on this benchmark.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "click accuracy",
   "unit": "%"
  },
  "name": "ScreenSpot-Pro",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2504.07981",
   "title": "ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use",
   "url": "https://arxiv.org/abs/2504.07981",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Kaixin Li",
    "Ziyang Meng",
    "Hongzhan Lin",
    "Ziyang Luo",
    "Yuchen Tian",
    "Jing Ma",
    "Zhiyong Huang",
    "Tat-Seng Chua"
   ],
   "org": "Independent research collaboration (Hong Kong Baptist University and collaborators)",
   "url": "https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding"
  },
  "released": "2025-01",
  "repo_url": "https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding",
  "saturation": {
   "as_of": "2026-08",
   "note": "At release, the best existing GUI grounding model (OS-Atlas-7B) scored 18.9%, and the paper's own proposed search method (ScreenSeekeR) reached 48.1% without additional training. The project's own leaderboard, fetched directly for this page, lists its current top entry at 82.7% (a \"zoom-in\" variant of a specialized grounding model, not a general frontier chat model), dated by the leaderboard page's own \"last updated\" notice rather than a specific submission date. Scores remain well below the ceiling and continue to separate models meaningfully.\n",
   "status": "open",
   "top_score": 82.7
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use (arXiv:2504.07981)",
    "url": "https://arxiv.org/abs/2504.07981"
   },
   {
    "accessed": "2026-09-08",
    "title": "likaixin2000/ScreenSpot-Pro-GUI-Grounding (readme.md, LICENSE)",
    "url": "https://github.com/likaixin2000/ScreenSpot-Pro-GUI-Grounding"
   },
   {
    "accessed": "2026-09-08",
    "title": "likaixin/ScreenSpot-Pro dataset card",
    "url": "https://huggingface.co/datasets/likaixin/ScreenSpot-Pro"
   },
   {
    "accessed": "2026-09-08",
    "title": "ScreenSpot-Pro Leaderboard",
    "url": "https://gui-agent.github.io/grounding-leaderboard/"
   }
  ],
  "status": "active",
  "subcategory": "GUI grounding",
  "summary": "1,581 instructions testing whether a model can point to the right UI element in authentic, high-resolution screenshots of 23 professional applications.",
  "tags": [
   "agentic",
   "gui-grounding",
   "computer-use",
   "multimodal",
   "professional-software"
  ],
  "task_format": "A screenshot plus a natural-language instruction describing an action (\"open the layers panel\", for example); the model outputs a single point or coordinate. Targets are additionally labelled as either text or icon.\n"
 }
}