SWE-bench Multilingual

SWE-bench Multilingual extends SWE-bench's real-GitHub-issue patch task to 300 tasks across 9 non-Python languages and 42 repositories.

Also known as: SWE-bench_Multilingual

unassessed

What it measures

SWE-bench Multilingual tests whether a model can resolve a real GitHub issue by patching a repository written in a language other than Python, since the original SWE-bench is Python-only. Each of the 300 task instances gives the model an issue description and a snapshot of one of 42 repositories, spanning C, C++, Go, Java, JavaScript, TypeScript, PHP, Ruby and Rust, at the commit before a real pull request fixed the issue. The model must locate the relevant code in an unfamiliar, non-Python codebase and produce a change a maintainer would accept, without being shown the tests that grade it.

Task format

Identical in spirit to SWE-bench: given an issue description and repository access, the system outputs a patch, applied inside a container and graded against FAIL_TO_PASS and PASS_TO_PASS tests recovered from the original fixing pull request. Multilingual reuses SWE-bench's dataset format and evaluation protocol so existing SWE-bench infrastructure can run it with no changes, but (unlike the original) does not pre-build shared dependency-cache "environment" images, since its 300 tasks are spread across 42 largely dependency-independent repositories rather than concentrated in a few.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub