{
 "body": "## What it measures\n\nSWE-EVO tests whether a coding agent can execute a realistic software evolution task instead of a\nsingle, isolated bug fix. Each task starts from the release notes of a mature open-source Python\nproject and asks the agent to make the set of multi-file changes the notes describe, working from a\nrepository snapshot taken before those changes landed.\n\nWhere SWE-bench style benchmarks isolate one GitHub issue in one or a few files, SWE-EVO tasks span\nan average of about 21 files and are checked against an average of 874 tests per instance, so success\nrequires understanding how a change ripples across a codebase rather than patching a single\nfunction.\n\n## How it is scored\n\nThe paper reports two metrics: Resolved rate, the share of task instances where all tests pass after\nthe agent's changes are applied, and Fix rate, a partial-credit metric giving the fraction of the\ntask's previously-failing tests that the agent's patch turns passing. Fix rate is intended to\ndistinguish agents that make real partial progress on a large task from agents that make none, since\nbinary resolution is rare at this task size. No random or human baseline is established in the\nsources reviewed.\n\n## Dataset and licence\n\nSWE-EVO comprises 48 tasks drawn from seven mature open-source Python projects, built from each\nproject's release notes rather than from individual issues. The reviewed sources do not state a\ndataset licence; this is left unknown rather than assumed. The benchmark's reference materials are\nhosted at the SWE-EVO GitHub repository.\n\n## Who publishes it\n\nSWE-EVO was introduced by researchers at the FPT Software AI Center, with a co-author from the\nUniversity of Melbourne's School of Computing and Information Systems. The paper, \"SWE-EVO:\nBenchmarking Coding Agents in Long-Horizon Software Evolution Scenarios,\" is authored by Minh Vu\nThai Pham, Tue Le, Dung Nguyen Manh, Huy Nhat Phan, and Nghi D. Q. Bui. No separate public leaderboard\nwas found in the sources reviewed.\n\n## Lineage\n\nSWE-EVO is positioned as a harder, longer-horizon successor to the single-issue SWE-bench family: the\npaper contrasts its own results directly against SWE-Bench Verified scores for the same models. It\ndoes not use SWE-bench's dataset or task format, so it is recorded here as a distinct benchmark in the\nSWE-bench family rather than a subset. No predecessor or successor benchmark within this repository\nwas established beyond that comparison.\n\n## Saturation and contamination\n\nSWE-EVO is not saturated: the paper reports GPT-5.4 with the OpenHands scaffold resolving only 25% of\ntasks, against 72.80% for the same class of model on SWE-Bench Verified, and frames this gap as\nevidence that current agents struggle with sustained, multi-file reasoning. Contamination risk is\nunknown; the reviewed sources do not describe a contamination check, and the tasks are built from\npublic release notes of existing open-source projects, which could appear in training data.\n\n## How to run it\n\nThere is no listed lm-evaluation-harness, HELM, or OpenCompass task name in the sources reviewed. The\npaper's own results were produced using the OpenHands agent scaffold, and a reference implementation\nis published at the SWE-EVO GitHub repository. Because the benchmark grades agents rather than raw\ncompletions, reported scores depend heavily on the agent scaffold, tool access, and step budget used,\nnone of which are standardized across reporters in the material reviewed.\n\n## Reading the numbers\n\nA high SWE-EVO score means an agent can plan and execute changes that span many files in a real\ncodebase and keep a large test suite passing, a stronger and rarer signal than resolving a single\nGitHub issue. A low score does not mean the agent is generally weak at coding, since the task horizon\nhere is intentionally much longer than SWE-bench's. Because Fix rate gives partial credit, look at\nboth Fix rate and Resolved rate together rather than either alone. With only 48 tasks, differences of\na few tasks translate into large percentage swings, so treat close scores as within noise. Compare\nscores only across the same agent scaffold and step budget, since the paper reports very different\nresults for the same model family depending on how it is deployed.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_evo",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "Tasks are built from real project release notes; the paper does not state a contamination study or training-data exclusion method, so exposure risk is not established.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "Python"
   ],
   "license": "",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": null,
   "size": 48,
   "size_note": "48 software-evolution tasks built from the release notes of 7 mature open-source Python projects; tasks average ~21 files changed and 874 tests per instance.",
   "splits": "single evaluation set",
   "url": "https://github.com/SWE-EVO/SWE-EVO"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Reference harness in the SWE-EVO GitHub repository; the paper reports results using OpenHands as the agent scaffold."
  },
  "id": "swe_evo",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench_verified",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-EVO tests whether a coding agent can carry out a realistic software evolution task rather than a single isolated bug fix. Each task is derived from the release notes of a mature open-source Python project and requires locating and editing code across many files to implement the described set of changes, then passing the project's real test suite. This targets sustained, multi-file reasoning over a large codebase rather than the single-issue, single-file patches typical of SWE-bench style benchmarks.\n",
  "metric": {
   "baseline_note": "Resolved rate is the share of tasks fully solved; Fix rate is a partial-credit metric based on the fraction of previously-failing tests the agent's patch makes pass.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "Resolved rate / Fix rate",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "SWE-EVO",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2512.18470",
   "title": "SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios",
   "url": "https://arxiv.org/abs/2512.18470",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Minh Vu Thai Pham",
    "Tue Le",
    "Dung Nguyen Manh",
    "Huy Nhat Phan",
    "Nghi D. Q. Bui"
   ],
   "org": "FPT Software AI Center; University of Melbourne (School of Computing and Information Systems)",
   "url": "https://github.com/SWE-EVO/SWE-EVO"
  },
  "released": "2026-04",
  "repo_url": "https://github.com/SWE-EVO/SWE-EVO",
  "saturation": {
   "as_of": "2026-04",
   "note": "GPT-5.4 with OpenHands scores 25% resolved on SWE-EVO versus 72.80% on SWE-Bench Verified, per the paper; the authors present this gap as evidence the benchmark is unsaturated and separates models that look similar on single-issue SWE-bench tasks.",
   "status": "open",
   "top_score": 25
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios",
    "url": "https://arxiv.org/abs/2512.18470"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-EVO paper, HTML rendering (v5)",
    "url": "https://arxiv.org/html/2512.18470v5"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-EVO reference repository",
    "url": "https://github.com/SWE-EVO/SWE-EVO"
   }
  ],
  "status": "active",
  "subcategory": "long-horizon software evolution / multi-file patch generation",
  "summary": "SWE-EVO gives coding agents a real project release note and asks for the multi-file changes it describes, checked against the project's own tests.",
  "tags": [
   "benchmark",
   "coding",
   "agentic",
   "multi-file",
   "software-evolution"
  ],
  "task_format": "The agent is given a repository snapshot and a natural-language description of the intended evolution (drawn from the project's release notes), and must produce a set of file edits. The result is graded by running the project's test suite, which averages 874 tests per task instance.\n"
 }
}