{
 "body": "\n## What it measures\n\nSWE-bench measures whether a language model can act as a software engineer on a real codebase: given a\nGitHub issue and the repository at the commit before it was fixed, produce a change that resolves it.\nEach of the 2,294 task instances is built from an actual merged pull request in one of 12 popular Python\nprojects (including Django, Sympy, scikit-learn and matplotlib), so tasks require finding the right\nfiles in a large, unfamiliar codebase and making a change consistent with how the project is actually\nstructured, not just solving an isolated puzzle. The model sees only the issue text; it does not see the\ntests that will grade it.\n\n## How it is scored\n\nEvery instance carries two sets of tests recovered from the original fixing pull request. FAIL_TO_PASS\ntests fail before the fix and pass after it; they are the primary signal that the issue was actually\nresolved. PASS_TO_PASS tests pass both before and after; they confirm the patch did not break unrelated\nfunctionality. A patch must satisfy both sets to count as resolved \u2014 there is no partial credit. The\noriginal paper's baseline used retrieval-augmented generation (RAG) to fetch candidate files before\nasking a model to generate a patch in one shot; most current results instead come from agentic systems\nthat read, run and edit the repository over multiple steps.\n\n## Dataset and licence\n\nThe dataset and evaluation code are released under the MIT licence. It contains 2,294 instances mined by\ncrawling pull requests and issues from 12 popular Python repositories, keeping only PRs that resolve an\nissue and touch at least one test file. Each instance ships with a Docker image that reproduces the\nrepository's environment at the pre-fix commit. All instances, and their tests, are public.\n\n## Who publishes it\n\nSWE-bench comes from Carlos E. Jimenez and John Yang (equal contribution), with Alexander Wettig, Shunyu\nYao, Kexin Pei, Ofir Press and Karthik Narasimhan, originally at Princeton NLP. It was released in\nOctober 2023 and accepted as an oral presentation at ICLR 2024. The project, now presented as maintained\nby \"the SWE-bench Team,\" continues to host the dataset, leaderboard and a growing family of related\ntools (SWE-agent, SWE-smith, SWE-ReX, mini-SWE-agent, the SWE-bench CLI) at swebench.com.\n\n## Lineage\n\nSWE-bench is the root of its own family. SWE-bench Verified (`swe_bench_verified`), released by OpenAI\nwith the SWE-bench authors in August 2024, explicitly supersedes both the original set and SWE-bench\nLite (a smaller, faster-to-run subset that does not yet have its own page in this repository) and has\nbecome the default reference. SWE-bench Multimodal (`swe_bench_multimodal`) and SWE-bench Multilingual\n(`swe_bench_multilingual`) extend the same collection methodology to visual, JavaScript/TypeScript tasks\nand to nine non-Python languages respectively. SWE-bench Pro (`swe_bench_pro`) is a harder, independently\nproduced benchmark from Scale AI in the same spirit rather than an official successor. `swe_bench_agent`\nis catalogued in this repository as a further family member; see its own page for what could and could\nnot be established about it.\n\n## Saturation and contamination\n\nOn the original, unfiltered instance set, OpenAI reported roughly 20% resolved for the best agents as of\nAugust 2024 \u2014 up sharply from a 1.96% baseline at launch and 12.47% for the first agentic system,\nSWE-agent, but still far from any ceiling. Contamination risk is high and structural: every instance is a\nreal pull request with public history, so a model trained after an instance's merge date may have seen\nthe literal fix rather than having to derive it. OpenAI's own investigation (leading to SWE-bench\nVerified) additionally found that many original instances were unfairly hard or impossible regardless of\ncontamination, due to underspecified issues or overly strict tests \u2014 a separate, non-contamination source\nof noise in reported scores.\n\n## How to run it\n\nThe reference harness lives in the SWE-bench GitHub repository and evaluates predictions inside\nper-instance Docker containers, applying a candidate patch and running the FAIL_TO_PASS/PASS_TO_PASS\ntests. It is invoked through the `swebench` command-line tool. Numbers are hard to compare across\npublishers because of differences in scaffold (RAG-style single-shot generation versus a multi-step\nagent with a shell), how much of the repository the system is allowed to see, and time/step budgets;\nnone of these are fixed by the benchmark itself.\n\n## Reading the numbers\n\nA \"SWE-bench\" score without qualification is ambiguous today: check whether it is the original 2,294-\ninstance set, SWE-bench Lite, or (most likely for a current model) SWE-bench Verified, since scores on\nthese are not interchangeable. A high score demonstrates a system can localise and fix real, moderately\nscoped bugs in familiar open-source Python projects; it says little about unfamiliar codebases, other\nlanguages, or issues requiring architectural changes, which is exactly the gap the later family members\nwere built to probe.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-bench Full",
   "SWE-bench Original"
  ],
  "category": "coding",
  "contamination": {
   "note": "Every task instance is a real, merged GitHub pull request with a public commit history; a model trained after an instance's fix was merged may have seen the exact patch during pretraining, not just the surrounding repository. This is a stronger contamination vector than \"the repo appears in training data\" alone.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "Python"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 2294,
   "size_note": "2,294 task instances collected from pull requests and issues across 12 popular Python repositories.",
   "splits": "single test split (no official train/dev split in the original release)",
   "url": "https://huggingface.co/datasets/princeton-nlp/SWE-bench"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Official Docker-based evaluation harness in the SWE-bench repository (see docs/20240627_docker), invoked via the `swebench` CLI (`swebench eval ...`). Not confirmed to be part of the lm-evaluation-harness, HELM, OpenCompass or BIG-bench task lists.\n"
  },
  "id": "swe_bench",
  "last_updated": "",
  "leaderboard_url": "https://www.swebench.com/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "swe_bench_verified"
   ],
   "variants": [
    "swe_bench_pro",
    "swe_bench_multilingual",
    "swe_bench_multimodal",
    "swe_bench_agent"
   ]
  },
  "measures": "SWE-bench gives a model a real GitHub issue and a snapshot of the repository at the commit before it was fixed, and asks it to produce a patch that resolves the issue. Tasks require locating the relevant code across a real, multi-file Python codebase, understanding what the issue is asking for, and making a change that a project maintainer would accept, rather than answering a self-contained puzzle. The model is not shown the tests used to grade it.\n",
  "metric": {
   "baseline_note": "No formal human baseline was established by the original paper.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved (FAIL_TO_PASS and PASS_TO_PASS tests both pass)",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench",
  "page_kind": "family",
  "paper": {
   "arxiv": "2310.06770",
   "title": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?",
   "url": "https://arxiv.org/abs/2310.06770",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Carlos E. Jimenez",
    "John Yang",
    "Alexander Wettig",
    "Shunyu Yao",
    "Kexin Pei",
    "Ofir Press",
    "Karthik R. Narasimhan"
   ],
   "org": "Originally Princeton NLP; maintained today by the SWE-bench Team",
   "url": "https://www.swebench.com/"
  },
  "released": "2023-10",
  "repo_url": "https://github.com/SWE-bench/SWE-bench",
  "saturation": {
   "as_of": "2024-08",
   "note": "OpenAI's SWE-bench Verified announcement cites 20% resolved on the original SWE-bench leaderboard and 43% on SWE-bench Lite as of 2024-08-05, up from a 1.96% RAG baseline at the October 2023 release and 12.47% for the first agent-based system (SWE-agent) shortly after. Community leaderboard activity on the unfiltered original set largely stopped after SWE-bench Verified launched days later, so a more recent top score for this exact dataset was not found.\n",
   "status": "open",
   "top_score": 20.0
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?",
    "url": "https://arxiv.org/abs/2310.06770"
   },
   {
    "accessed": "2026-09-07",
    "title": "SWE-bench/SWE-bench repository",
    "url": "https://github.com/SWE-bench/SWE-bench"
   },
   {
    "accessed": "2026-09-07",
    "title": "SWE-bench (original) overview",
    "url": "https://www.swebench.com/original.html"
   },
   {
    "accessed": "2026-09-07",
    "title": "SWE-bench project overview",
    "url": "https://www.swebench.com/SWE-bench/"
   },
   {
    "accessed": "2026-09-07",
    "title": "Introducing SWE-bench Verified | OpenAI",
    "url": "https://openai.com/index/introducing-swe-bench-verified/"
   }
  ],
  "status": "superseded",
  "subcategory": "GitHub issue resolution / patch generation",
  "summary": "SWE-bench tests whether a model can resolve real GitHub issues by generating a patch, checked by the repository's own test suite; Verified now supersedes the original set for most reporting.",
  "tags": [
   "coding",
   "agentic",
   "github-issues",
   "patch-generation",
   "docker",
   "software-engineering"
  ],
  "task_format": "Given an issue description and repository access (directly, or through retrieval, depending on the system under test), the model outputs a patch/diff. The patch is applied to a containerised checkout of the repository and graded by running two sets of tests recovered from the pull request that originally fixed the issue.\n"
 }
}