{
 "body": "## What it measures\n\nSWE-bench Extra follows the same task design as SWE-bench: a system is given a real GitHub issue and\na snapshot of the repository at the commit before the fix, and must produce a patch that resolves the\nissue, checked against the project's own tests. Rather than adding a new task format, it extends the\npool of such tasks beyond SWE-bench's original 12 repositories, drawing issue-and-pull-request pairs\nfrom 1,988 distinct Python repositories.\n\nThe dataset card describes it as intended \"to train or evaluate agentic systems specializing in\nresolving GitHub issues,\" which positions it as a larger and more diverse pool of SWE-bench-style\ntasks for training or broad evaluation rather than a fixed, leaderboard-graded benchmark release.\n\n## How it is scored\n\nScoring follows the SWE-bench convention when used for evaluation: a patch is applied to a\ncontainerised checkout and the instance counts as resolved only if the FAIL_TO_PASS tests recovered\nfrom the original pull request now pass and the PASS_TO_PASS tests continue to pass. No random or\nhuman baseline, and no published leaderboard score, was found in the sources reviewed.\n\n## Dataset and licence\n\nThe dataset contains 6,415 issue-and-pull-request pairs collected across 1,988 Python repositories,\npublished under the CC-BY-4.0 licence. The Hugging Face release lists a single \"train\" split of 6,376\nexamples, roughly 88MB uncompressed. Each example includes fields such as instance ID, base commit,\npatch, repository name, and problem statement, matching the SWE-bench instance schema.\n\n## Who publishes it\n\nSWE-bench Extra is published by Nebius on Hugging Face. No accompanying paper or named author list was\nfound in the sources reviewed, and the dataset card does not identify a maintained leaderboard.\n\n## Lineage\n\nSWE-bench Extra is built using the SWE-bench methodology and instance schema, so it is recorded here\nin the SWE-bench family with SWE-bench itself as its predecessor. It is not a subset of SWE-bench's\nown repositories: it collects pairs from a much larger and different pool of 1,988 repositories, so it\nshould be treated as an independent, larger dataset in the same style rather than a sample of the\noriginal benchmark.\n\n## Saturation and contamination\n\nNo leaderboard or reported model scores were found in the sources reviewed, so saturation status is\nunknown. Contamination risk is also unknown: the underlying issues and pull requests are public\nGitHub content in the same style as SWE-bench, which has a documented risk of appearing in training\ndata, but the dataset card does not state a decontamination process for this specific collection.\n\n## How to run it\n\nNo lm-evaluation-harness, HELM, or OpenCompass task name was found in the sources reviewed. The\ndataset card indicates it follows the SWE-bench evaluation methodology (patch application inside a\ncontainer, graded by the project's recovered tests), so the reference SWE-bench harness is the\npractical way to run it, though this was not explicitly confirmed as a packaged task in that harness.\n\n## Reading the numbers\n\nBecause SWE-bench Extra has no established leaderboard in the sources reviewed, a reported score\nshould be read as a self-reported result on this specific 6,415-instance pool rather than a comparison\nagainst other models' scores. A high resolve rate indicates the system can locate and fix real issues\nacross a wide variety of unfamiliar Python repositories, which is a broader test of generalisation than\nSWE-bench's original 12 repositories. Results are not directly comparable to SWE-bench or SWE-bench\nVerified scores, since the task pool, and therefore task difficulty distribution, differs. Treat the\ndataset primarily as a training and evaluation resource rather than a standardized benchmark until a\nleaderboard or comparative study is published.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_extra",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "Built from real, public GitHub issues and pull requests in the SWE-bench style; the dataset card does not state a decontamination or training-data exclusion process.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "Python"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": null,
   "size": 6415,
   "size_note": "6,415 issue-and-pull-request pairs across 1,988 Python repositories; the published Hugging Face split totals 6,376 examples (~88MB uncompressed, ~25MB download).",
   "splits": "single 'train' split",
   "url": "https://huggingface.co/datasets/nebius/SWE-bench-extra"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Uses the SWE-bench task/container evaluation methodology per the dataset card; no specific harness task name was found."
  },
  "id": "swe_bench_extra",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-bench Extra follows the SWE-bench task design: given a real GitHub issue and a snapshot of the repository at the commit before the fix, a model or agent must produce a patch that resolves the issue, verified by the project's own tests. It extends the pool of such tasks well beyond the original SWE-bench's 12 repositories, drawing issue-and-pull-request pairs from 1,988 Python repositories, and is positioned for training and evaluating agentic systems that resolve GitHub issues rather than as a fixed leaderboard benchmark.\n",
  "metric": {
   "baseline_note": "No baseline or leaderboard was found in the sources reviewed; the dataset card presents it as a training/evaluation resource rather than a scored competition.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved (FAIL_TO_PASS and PASS_TO_PASS tests pass)",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "SWE-bench Extra",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "Nebius",
   "url": "https://huggingface.co/datasets/nebius/SWE-bench-extra"
  },
  "released": "",
  "repo_url": "https://huggingface.co/datasets/nebius/SWE-bench-extra",
  "saturation": {
   "as_of": "",
   "note": "No scored leaderboard or reported model results were found in the sources reviewed.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "nebius/SWE-bench-extra dataset card",
    "url": "https://huggingface.co/datasets/nebius/SWE-bench-extra"
   },
   {
    "accessed": "2026-09-08",
    "title": "nebius/SWE-bench-extra dataset metadata (API)",
    "url": "https://huggingface.co/api/datasets/nebius/SWE-bench-extra"
   }
  ],
  "status": "active",
  "subcategory": "GitHub issue resolution / patch generation",
  "summary": "SWE-bench Extra is a 6,415-instance dataset of real GitHub issue-and-fix pairs, built with the SWE-bench methodology to extend beyond the original benchmark's repositories.",
  "tags": [
   "benchmark",
   "coding",
   "agentic",
   "github-issues"
  ],
  "task_format": "Given an issue description and a repository checkout, the system produces a patch/diff, applied to a container and graded against tests recovered from the pull request that originally closed the issue, following the SWE-bench protocol.\n"
 }
}