SWE-bench Multimodal tests GitHub-issue patching on JavaScript/TypeScript repositories where the issue includes an image, such as a bug screenshot or design mockup.
unassessed
| Category | coding |
|---|---|
| Subcategory | GitHub issue resolution / patch generation with visual inputs |
| Page status | active |
| Metric | % resolved (FAIL_TO_PASS and PASS_TO_PASS tests both pass) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 480 |
| Dataset licence | CC BY 4.0 |
| Publisher | Originally Princeton NLP / Stanford, with the SWE-bench team; maintained today by the SWE-bench Team at swebench.com |
SWE-bench Multimodal measures the same underlying skill as SWE-bench — resolving a real GitHub issue with a patch a project's own tests accept — but restricted to issues that include a visual element: a screenshot of a bug, a design mockup or wireframe, a diagram of the desired behaviour, or an error message with visual context. Repositories are JavaScript/TypeScript projects for web interface design, diagramming, data visualisation, syntax highlighting and interactive mapping, so a model must read and act on both text and images to succeed, not just text.
Given an issue description that includes at least one image, plus repository access, the system outputs a patch. The patch is applied inside a container and graded against FAIL_TO_PASS and PASS_TO_PASS tests recovered from the pull request that originally fixed the issue, the same protocol as the rest of the SWE-bench family.
| Model | Provider | Score | Card as of |
|---|---|---|---|
| Claude Mythos Preview | Anthropic | 59.0 | 2026-04 |
| Claude Opus 4.6 | Anthropic | 27.1 | 2026-04 |