SWE-EVO gives coding agents a real project release note and asks for the multi-file changes it describes, checked against the project's own tests.
unassessed
| Category | coding |
|---|---|
| Subcategory | long-horizon software evolution / multi-file patch generation |
| Page status | active |
| Metric | Resolved rate / Fix rate |
| Direction | higher_is_better |
| Unit | percent |
| Dataset size | 48 |
| Publisher | FPT Software AI Center; University of Melbourne (School of Computing and Information Systems) |
SWE-EVO tests whether a coding agent can carry out a realistic software evolution task rather than a single isolated bug fix. Each task is derived from the release notes of a mature open-source Python project and requires locating and editing code across many files to implement the described set of changes, then passing the project's real test suite. This targets sustained, multi-file reasoning over a large codebase rather than the single-issue, single-file patches typical of SWE-bench style benchmarks.
The agent is given a repository snapshot and a natural-language description of the intended evolution (drawn from the project's release notes), and must produce a set of file edits. The result is graded by running the project's test suite, which averages 874 tests per task instance.
No model card in ModelSpec reports this benchmark yet.