Scale AI's harder, contamination-resistant successor in spirit to SWE-bench Verified: 1,865 long-horizon tasks across public copyleft, held-out and private commercial codebases.
unassessed
| Category | coding |
|---|---|
| Subcategory | GitHub issue resolution / patch generation |
| Page status | active |
| Metric | % resolved |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1865 |
| Dataset licence | MIT |
| Publisher | Scale AI (Scale Research Team) |
SWE-bench Pro follows the same task shape as SWE-bench — resolve a real issue in a real repository with a patch — but selects for longer, more involved changes and for source code that is unlikely to be in a model's training data. Scale AI built it because frontier models were already scoring above 70% on SWE-bench Verified, and wanted a benchmark that separates genuine software-engineering ability from memorised or near-memorised solutions on well-known open-source Python projects.
Given an issue and repository access, the system produces a patch, applied and graded inside a container against the repository's own tests, in the same fail-to-pass / pass-to-pass style as SWE-bench. Tasks are deliberately long-horizon: the paper reports resolved instances require changes averaging 107.4 lines of code across 4.1 files, more than a typical SWE-bench Verified fix.
| Model | Provider | Score | Card as of |
|---|---|---|---|
| Claude Mythos Preview | Anthropic | 77.8 | 2026-04 |
| GLM 5.1 | Z.ai (Zhipu AI) | 58.4 | 2026-04 |
| GPT-5.4 | OpenAI | 57.7 | 2026-04 |
| Gemini 3.1 Pro Preview | Google DeepMind | 54.2 | 2026-04 |
| Claude Opus 4.6 | Anthropic | 53.4 | 2026-04 |