SWE-bench Verified

A 500-task, human-screened subset of SWE-bench that OpenAI released with the SWE-bench authors to remove unfair or impossible samples; now the default SWE-bench reference.

Also known as: SWE-bench-V

unassessed

What it measures

SWE-bench Verified measures the same thing as SWE-bench: whether a model can resolve a real GitHub issue by patching a Python repository, graded by the tests from the pull request that actually fixed it. The difference is curation: OpenAI ran a large human-annotation campaign to remove instances whose issue description was too vague or whose tests would reject a genuinely correct fix, so a low score is more likely to reflect a real capability gap than a broken task.

Task format

Identical to SWE-bench: given an issue description and repository access, the system outputs a patch, which is applied inside a container and graded against FAIL_TO_PASS and PASS_TO_PASS tests recovered from the original fixing pull request.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub