SWE-bench tests whether a model can resolve real GitHub issues by generating a patch, checked by the repository's own test suite; Verified now supersedes the original set for most reporting.
unassessed
| Category | coding |
|---|---|
| Subcategory | GitHub issue resolution / patch generation |
| Page status | superseded |
| Metric | % resolved (FAIL_TO_PASS and PASS_TO_PASS tests both pass) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2294 |
| Dataset licence | MIT |
| Publisher | Originally Princeton NLP; maintained today by the SWE-bench Team |
SWE-bench gives a model a real GitHub issue and a snapshot of the repository at the commit before it was fixed, and asks it to produce a patch that resolves the issue. Tasks require locating the relevant code across a real, multi-file Python codebase, understanding what the issue is asking for, and making a change that a project maintainer would accept, rather than answering a self-contained puzzle. The model is not shown the tests used to grade it.
Given an issue description and repository access (directly, or through retrieval, depending on the system under test), the model outputs a patch/diff. The patch is applied to a containerised checkout of the repository and graded by running two sets of tests recovered from the pull request that originally fixed the issue.
No model card in ModelSpec reports this benchmark yet.