{
 "body": "\n## What it measures\n\nSWE-bench-Live tests whether an agent can fix a recent, real GitHub issue. The agent\nsees the issue text and a containerized checkout from before the fix, then writes a\npatch. Tests recovered from the fixing pull request decide whether the issue is\nresolved. The original paper set is Python. The same project later added MultiLang\nand Windows tasks that follow the same issue-and-patch shape.\n\nThe point of the \"live\" design is freshness and coverage, not a new scoring rule.\nRepoLaunch builds an executable image per instance so new repositories can enter\nthe pool without months of hand-written environments.\n\n## How it is scored\n\nScoring copies SWE-bench. FAIL_TO_PASS tests must start failing and end passing.\nPASS_TO_PASS tests must stay passing. There is no partial credit. The paper also\nreports whether the patch applies cleanly and whether edited files match the gold\npatch, but resolved rate is the headline number.\n\nProtocol restrictions are strict. The agent may not apply the hidden test patch\nduring rollout, and prompts must not contain instance-specific solutions. The\nmaintainers ask for raw trajectories with leaderboard submissions so they can check\nthat rule.\n\n## Dataset and licence\n\nThe code and Python dataset are MIT-licensed. Hugging Face currently publishes 1,888\nPython instances in `full`, with frozen `lite` (300) and `verified` (500) splits\nkept for cheaper, comparable runs. The May 2025 paper counted 1,319 Python tasks\nfrom 93 repositories. From 2025-09 the maintainers add about 50 newly filtered\nissues a month to the growing pool (`full` on Hugging Face; GitHub's note names\n`test`). MultiLang adds 1,077 instances across eight language splits. Windows is a\nseparate 61-instance set. Gold patches and test lists ship in the dataset, but the\npublished protocol forbids using them at generation time.\n\n## Who publishes it\n\nMicrosoft researchers led by Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, and\ncolleagues released the paper in May 2025 (arXiv:2505.23419). The GitHub repository\ntitles the work as NeurIPS 2025 Datasets and Benchmarks and cites NeurIPS volume 38.\nThe repository, Hugging Face collection, and leaderboard remain under SWE-bench-Live.\nA later paper, RepoLaunch (arXiv:2603.05026, submitted 2026-03-05), describes the\nmulti-language and Windows environment builder.\n\n## Lineage\n\nThis is a SWE-bench family member (`swe_bench`), not a replacement for\n[SWE-bench Verified](swe_bench_verified.md). It keeps the same fail-to-pass tests\nwhile changing how instances are collected and how often they refresh. Do not\nconfuse its frozen 300-instance `lite` split with official\n[SWE-bench Lite](swe_bench_lite.md). LiveCodeBench is an unrelated live coding\ncontest set. Multi-SWE-bench and SWE-bench Multilingual are separate multilingual\npools.\n\n## Saturation and contamination\n\nOn the paper's full Python set, the best reported agent resolved 19.25% of tasks,\nversus 43.20% for the same OpenHands plus Claude 3.7 Sonnet run on Verified. That\ngap is the reason the authors argue static SWE-bench numbers overstate skill on\nunseen issues. Scores still separate models, so the set is open rather than\nsaturated. Contamination risk is medium: public history remains, but post-2024\nissues and monthly additions reduce overlap with older training cuts.\n\n## How to run it\n\nInstall the official repository and run its evaluation script on a predictions\ndirectory. For the Python NeurIPS set, follow the `python-only` branch. For\nMultiLang or Windows, use `python -m evaluation.evaluation` on the matching\nHugging Face dataset. Compare only like splits: Lite, Verified, full, MultiLang,\nor Windows. Scaffold, iteration budget, and whether trajectories were audited all\nmove the number.\n\n## Reading the numbers\n\nA strong Live score means the system can localise and patch recent issues in\nrepositories that were not in the original twelve-project SWE-bench set. It does\nnot mean the same resolved rate will hold on Verified or Lite. Always name the\nsplit and date, because `full` grows. Treat a large jump on a frozen split as a\ncapability claim, and a jump only on newly added months as a freshness check.\nLook at Verified or SWE-bench Pro alongside it when you need a stable, heavily\nreported reference.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_live",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-bench Live",
   "SWE-bench Goes Live"
  ],
  "category": "coding",
  "contamination": {
   "note": "Instances are real public pull requests, so later training runs can still see gold patches. Recency (issues after 2024) and monthly additions lower that risk relative to the 2023 SWE-bench pool. Frozen lite and verified splits age in place; the full split is the moving target.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "Python",
    "C",
    "C++",
    "C#",
    "Java",
    "JavaScript",
    "TypeScript",
    "Go",
    "Rust"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 1888,
   "size_note": "Hugging Face SWE-bench-Live/SWE-bench-Live (Python, accessed 2026-09-09) lists four splits: test 1,000, lite 300, verified 500, and full 1,888. The May 2025 paper described an initial 1,319 Python tasks from 93 repositories (issues from 2024-01 through 2025-04). Maintainers froze the Python `lite` and `verified` splits for leaderboard comparisons. From 2025-09 they add about 50 newly quality-filtered issues each month; the Hugging Face card sends those to `full`, while the 2025-09-17 GitHub note says they go to `test`. Related public sets from the same project are SWE-bench-Live/MultiLang (1,077 instances across C, C++, Go, JavaScript, Rust, Java, TypeScript, and C# as of 2026-09-04) and SWE-bench-Live/Windows (61 test instances, MIT, last modified 2026-09-04).\n",
   "splits": "Python: test 1000, lite 300, verified 500, full 1888; MultiLang language splits; Windows test 61",
   "url": "https://huggingface.co/datasets/SWE-bench-Live/SWE-bench-Live"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-074 (Codex coordinated)",
   "reviewed": "2026-09-09",
   "reviewed_by": "Grok Build independent review, batch-074"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Official evaluation lives in microsoft/SWE-bench-Live. Maintainers recommend the python-only branch for the NeurIPS Python set, and the main-branch evaluation.evaluation entry point for MultiLang and Windows. Submissions go through SWE-bench-Live/submission with rollout trajectories.\n"
  },
  "id": "swe_bench_live",
  "last_updated": "2026-09",
  "leaderboard_url": "https://swe-bench-live.github.io/",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench",
   "successors": [],
   "variants": [
    "swe_bench_verified",
    "swe_bench_lite"
   ]
  },
  "measures": "SWE-bench-Live asks an agent to resolve a real GitHub issue on a snapshot of the repository from before the fix, then grades the patch with the project's tests. Unlike the original SWE-bench pool, instances are mined automatically from issues created since 2024 and refreshed over time so evaluation is less likely to rest on patches already seen in pretraining.\n",
  "metric": {
   "baseline_note": "No human solve-rate baseline is published. The paper also reports patch apply rate and file-level localization success.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench-Live",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.23419",
   "title": "SWE-bench Goes Live!",
   "url": "https://arxiv.org/abs/2505.23419",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Linghao Zhang",
    "Shilin He",
    "Chaoyun Zhang",
    "Yu Kang",
    "Bowen Li",
    "Chengxing Xie",
    "Junhao Wang",
    "Maoquan Wang",
    "Yufan Huang",
    "Shengyu Fu",
    "Elsie Nallipogu",
    "Qingwei Lin",
    "Yingnong Dang",
    "Saravan Rajmohan",
    "Dongmei Zhang"
   ],
   "org": "Microsoft",
   "url": "https://swe-bench-live.github.io/"
  },
  "released": "2025-05",
  "repo_url": "https://github.com/microsoft/SWE-bench-Live",
  "saturation": {
   "as_of": "2025-05",
   "note": "On the paper's then-current full Python set, OpenHands with Claude 3.7 Sonnet resolved 19.25% of instances (17.67% on the 300-instance Lite subset). The same setup resolved 43.20% on SWE-bench Verified. Later MultiLang and Windows leaderboard numbers were not extracted from the JavaScript homepage, so a 2026 top score is not recorded here.\n",
   "status": "open",
   "top_score": 19.25
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Goes Live! (arXiv:2505.23419)",
    "url": "https://arxiv.org/abs/2505.23419"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Goes Live! HTML full text",
    "url": "https://arxiv.org/html/2505.23419v2"
   },
   {
    "accessed": "2026-09-08",
    "title": "microsoft/SWE-bench-Live repository README",
    "url": "https://github.com/microsoft/SWE-bench-Live"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench-Live Python dataset card",
    "url": "https://huggingface.co/datasets/SWE-bench-Live/SWE-bench-Live"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench-Live/MultiLang dataset API",
    "url": "https://huggingface.co/api/datasets/SWE-bench-Live/MultiLang"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench-Live leaderboard homepage",
    "url": "https://swe-bench-live.github.io/"
   },
   {
    "accessed": "2026-09-09",
    "title": "SWE-bench-Live MIT licence",
    "url": "https://raw.githubusercontent.com/microsoft/SWE-bench-Live/main/LICENSE"
   },
   {
    "accessed": "2026-09-09",
    "title": "RepoLaunch (arXiv:2603.05026)",
    "url": "https://arxiv.org/abs/2603.05026"
   },
   {
    "accessed": "2026-09-09",
    "title": "SWE-bench-Live/Windows dataset API",
    "url": "https://huggingface.co/api/datasets/SWE-bench-Live/Windows"
   }
  ],
  "status": "active",
  "subcategory": "live GitHub issue resolution / patch generation",
  "summary": "A continuously updated SWE-bench-style issue-resolution set built from recent GitHub issues, with frozen Lite/Verified splits and a growing full split.",
  "tags": [
   "coding",
   "agentic",
   "github-issues",
   "patch-generation",
   "docker",
   "live",
   "contamination-resistant"
  ],
  "task_format": "The agent may use only the problem statement and the instance Docker image. It must not read FAIL_TO_PASS, PASS_TO_PASS, hints, or the test patch during rollout. A single patch is applied in the container and scored with fail-to-pass and pass-to-pass tests, following the original SWE-bench protocol.\n"
 }
}