A continuously updated SWE-bench-style issue-resolution set built from recent GitHub issues, with frozen Lite/Verified splits and a growing full split.
unassessed
| Category | coding |
|---|---|
| Subcategory | live GitHub issue resolution / patch generation |
| Page status | active |
| Metric | % resolved |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1888 |
| Dataset licence | MIT |
| Publisher | Microsoft |
SWE-bench-Live asks an agent to resolve a real GitHub issue on a snapshot of the repository from before the fix, then grades the patch with the project's tests. Unlike the original SWE-bench pool, instances are mined automatically from issues created since 2024 and refreshed over time so evaluation is less likely to rest on patches already seen in pretraining.
The agent may use only the problem statement and the instance Docker image. It must not read FAIL_TO_PASS, PASS_TO_PASS, hints, or the test patch during rollout. A single patch is applied in the container and scored with fail-to-pass and pass-to-pass tests, following the original SWE-bench protocol.
No model card in ModelSpec reports this benchmark yet.