{
 "body": "\n## What it measures\n\nSWE-bench Pro measures the same core skill as SWE-bench \u2014 turning a real issue into a patch a\nrepository's own tests accept \u2014 but is built specifically to resist the two failure modes Scale AI\nargues undermine SWE-bench Verified at the frontier: data contamination and tasks that are easier than\nreal engineering work. Instances require substantial, multi-file changes rather than small localized\nfixes, and are drawn from code that is legally or practically unlikely to have entered a model's\ntraining data.\n\n## How it is scored\n\nScoring follows the SWE-bench pattern: a generated patch is applied inside a container and graded by\nrunning tests derived from the original fix, with fail-to-pass tests confirming the issue is resolved\nand pass-to-pass tests confirming nothing else broke. Scale's own evaluation used the SWE-Agent scaffold\nto produce the headline numbers, and its released evaluation scripts support running the same protocol\nat scale with distributed workers.\n\n## Dataset and licence\n\nThe evaluation code is MIT-licensed. The dataset totals 1,865 instances across 41 repositories, but only\nthe 731-instance public subset (11 repositories, all under strong copyleft licences such as GPL) is\nopenly released; 858 further held-out instances and 276 private commercial instances (from 18 startups,\nunder formal partnership) are kept back specifically so they cannot leak into future training runs.\nTasks average 107.4 lines of code changed across 4.1 files, a materially larger footprint than a typical\nSWE-bench Verified fix.\n\n## Who publishes it\n\nSWE-bench Pro is a Scale AI Research Team project, led by Xiang Deng and Jeff Da with more than a dozen\nco-authors, first published 2025-09-19 (blog) and 2025-09-21 (arXiv paper), with a revised paper version\nin November 2025. Scale maintains the dataset, open-source evaluation repository and two separate\nleaderboards.\n\n## Lineage\n\nSWE-bench Pro is not part of the official swebench.com project; it is an independent benchmark from\nScale AI that explicitly positions itself as picking up \"where SWE-Bench Verified leaves off,\" reusing\nSWE-bench's task format and fail-to-pass/pass-to-pass scoring convention while sourcing new, harder,\ncontamination-resistant tasks. It is catalogued in this repository as a member of the `swe_bench` family\nalongside `swe_bench_verified`, `swe_bench_multilingual`, `swe_bench_multimodal` and `swe_bench_agent`.\n\n## Saturation and contamination\n\nThe benchmark is wide open: the best models resolve under a quarter of public-subset tasks, with a long\ntail of weaker models scoring in the single digits, and scores drop again on the private commercial\nsubset (for example Claude Opus 4.1 from roughly 23% to 17.8%, GPT-5 from roughly 23% to 14.9%, per\nScale's own reporting). Contamination risk is low by design \u2014 copyleft licensing for the public set,\nfull privacy for the commercial set \u2014 though that is the publisher's own claim about its methodology\nrather than something this page independently verified.\n\n## How to run it\n\nPublic-subset evaluation uses the open SWE-bench_Pro-os repository's Docker-based harness and evaluation\nscripts; the held-out and commercial subsets are only accessible through Scale's own leaderboard\nsubmission process, so those numbers cannot be independently reproduced by a third party. Because the\npublished results use a specific scaffold (SWE-Agent), scores from other harnesses are not guaranteed to\nbe comparable without checking which scaffold produced them.\n\n## Reading the numbers\n\nA SWE-bench Pro score in the 20s should be read as evidence of real difficulty separation at the current\nfrontier, not as a sign the benchmark is broken: it is designed to be hard, and its low scores relative\nto SWE-bench Verified are the point, not a bug. The gap between a model's public-subset and\ncommercial-subset scores is itself informative \u2014 a large gap suggests a model leans on some form of\nprior exposure or overfitting to public conventions rather than general code-repair skill.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_pro",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 77.8,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GLM 5.1",
   "model_id": "zhipu/glm-5-1",
   "provider": "zhipu",
   "provider_display": "Z.ai (Zhipu AI)",
   "score": 58.4,
   "source": "zai-org-model-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-5.4",
   "model_id": "openai/gpt-5-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 57.7,
   "source": "lmarena.ai, provider-reports, anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 3.1 Pro Preview",
   "model_id": "google/gemini-3-1-pro-preview",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 54.2,
   "source": "anthropic-system-card-mythos, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 53.4,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  }
 ],
 "page": {
  "aliases": [
   "SWE-Bench Pro"
  ],
  "category": "coding",
  "contamination": {
   "note": "Low by explicit design rather than by independent audit: the public subset is drawn from strong-copyleft-licensed repositories that Scale argues are unlikely to appear in training data for legal reasons, and the commercial subset is entirely private code obtained under partnership, unpublished anywhere a crawler could reach it. This reflects the publisher's stated design intent; it was not independently verified for this page.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "Go",
    "Python",
    "JavaScript",
    "TypeScript"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": false,
   "size": 1865,
   "size_note": "1,865 instances across 41 actively maintained repositories, split into three subsets: 731 public instances from 11 repositories released under strong copyleft licences (for example GPL), chosen because their licence terms make inclusion in training data legally unattractive; 858 held-out instances from 12 further public repositories, kept out of the open release; and 276 commercial instances from 18 private, proprietary startup codebases under formal partnership with Scale. Each repository contributes roughly 50-100 tasks.\n",
   "splits": "public (731, open), held-out (858, withheld), commercial (276, withheld)",
   "url": "https://huggingface.co/datasets/ScaleAI/SWE-bench_Pro"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The paper's own results use the SWE-Agent scaffold. Scale publishes evaluation code (swe_bench_pro_eval.py) and Docker-based test infrastructure in the SWE-bench_Pro-os repository, supporting distributed evaluation with configurable worker counts.\n"
  },
  "id": "swe_bench_pro",
  "last_updated": "2025-11",
  "leaderboard_url": "https://labs.scale.com/leaderboard/swe_bench_pro_public",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench_verified",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-bench Pro follows the same task shape as SWE-bench \u2014 resolve a real issue in a real repository with a patch \u2014 but selects for longer, more involved changes and for source code that is unlikely to be in a model's training data. Scale AI built it because frontier models were already scoring above 70% on SWE-bench Verified, and wanted a benchmark that separates genuine software-engineering ability from memorised or near-memorised solutions on well-known open-source Python projects.\n",
  "metric": {
   "baseline_note": "No human baseline published.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench Pro",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2509.16941",
   "title": "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
   "url": "https://arxiv.org/abs/2509.16941",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Xiang Deng",
    "Jeff Da",
    "Edwin Pan",
    "Yannis Yiming He",
    "Charles Ide",
    "Kanak Garg",
    "Niklas Lauffer",
    "Andrew Park",
    "Nitin Pasari",
    "Chetan Rane",
    "Karmini Sampath",
    "Maya Krishnan",
    "Srivatsa Kundurthy",
    "Sean Hendryx",
    "Zifan Wang",
    "Vijay Bharadwaj",
    "Jeff Holm",
    "Raja Aluri",
    "Chen Bo Calvin Zhang",
    "Noah Jacobson",
    "Bing Liu",
    "Brad Kenstler"
   ],
   "org": "Scale AI (Scale Research Team)",
   "url": "https://scale.com/research/swe_bench_pro"
  },
  "released": "2025-09",
  "repo_url": "https://github.com/scaleapi/SWE-bench_Pro-os",
  "saturation": {
   "as_of": "2025-09",
   "note": "At release, the best models (GPT-5 at 23.3%, Claude Opus 4.1 at 23.1%) resolved under a quarter of public-subset tasks, a steep drop from the 70%+ scores the same class of model reaches on SWE-bench Verified. Weaker or older models score far lower still (GPT-4o 4.9%, DeepSeek/Qwen-3 32B 3.4%), giving wide separation between models. Scores drop further on the private commercial subset, consistent with the benchmark's contamination-resistance goal. This is an early, wide-open benchmark, not a saturated one.\n",
   "status": "open",
   "top_score": 23.3
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "SWE-Bench Pro: Raising the Bar for Agentic Coding",
    "url": "https://scale.com/blog/swe-bench-pro"
   },
   {
    "accessed": "2026-09-07",
    "title": "SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?",
    "url": "https://arxiv.org/abs/2509.16941"
   },
   {
    "accessed": "2026-09-07",
    "title": "scaleapi/SWE-bench_Pro-os repository",
    "url": "https://github.com/scaleapi/SWE-bench_Pro-os"
   }
  ],
  "status": "active",
  "subcategory": "GitHub issue resolution / patch generation",
  "summary": "Scale AI's harder, contamination-resistant successor in spirit to SWE-bench Verified: 1,865 long-horizon tasks across public copyleft, held-out and private commercial codebases.",
  "tags": [
   "coding",
   "agentic",
   "github-issues",
   "patch-generation",
   "contamination-resistant",
   "long-horizon",
   "private-eval"
  ],
  "task_format": "Given an issue and repository access, the system produces a patch, applied and graded inside a container against the repository's own tests, in the same fail-to-pass / pass-to-pass style as SWE-bench. Tasks are deliberately long-horizon: the paper reports resolved instances require changes averaging 107.4 lines of code across 4.1 files, more than a typical SWE-bench Verified fix.\n"
 }
}