SWE-bench Extra is a 6,415-instance dataset of real GitHub issue-and-fix pairs, built with the SWE-bench methodology to extend beyond the original benchmark's repositories.
unassessed
| Category | coding |
|---|---|
| Subcategory | GitHub issue resolution / patch generation |
| Page status | active |
| Metric | % resolved (FAIL_TO_PASS and PASS_TO_PASS tests pass) |
| Direction | higher_is_better |
| Unit | percent |
| Dataset size | 6415 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Nebius |
SWE-bench Extra follows the SWE-bench task design: given a real GitHub issue and a snapshot of the repository at the commit before the fix, a model or agent must produce a patch that resolves the issue, verified by the project's own tests. It extends the pool of such tasks well beyond the original SWE-bench's 12 repositories, drawing issue-and-pull-request pairs from 1,988 Python repositories, and is positioned for training and evaluating agentic systems that resolve GitHub issues rather than as a fixed leaderboard benchmark.
Given an issue description and a repository checkout, the system produces a patch/diff, applied to a container and graded against tests recovered from the pull request that originally closed the issue, following the SWE-bench protocol.
No model card in ModelSpec reports this benchmark yet.