{
 "body": "\n## What it measures\n\nSWE-bench Multimodal asks whether a model's ability to resolve real GitHub issues, as measured by\nSWE-bench, holds up when the issue itself is partly visual. Every task instance's problem statement or\ntests include at least one image \u2014 a screenshot of a bug, a design mockup or wireframe, a diagram of\nintended behaviour, or an error message shown in context \u2014 drawn from JavaScript/TypeScript repositories\ncovering web interface design, diagramming, data visualisation, syntax highlighting and interactive\nmapping. A model must read and act on both the text and the image to produce a patch, not just the text.\n\n## How it is scored\n\nScoring follows the rest of the SWE-bench family exactly: a candidate patch is applied inside a container\nand must make the instance's FAIL_TO_PASS tests pass while leaving its PASS_TO_PASS tests passing, with no\npartial credit for a partially correct fix. What differs is the input and the target codebase, not the\ngrading mechanics.\n\n## Dataset and licence\n\nReleased under CC BY 4.0, hosted on Hugging Face. The original October 2024 release held 517 task\ninstances containing at least one image, mined from JavaScript/TypeScript projects using SWE-bench's usual\nGitHub issue/PR collection pipeline. A \"v2\" refresh, dated 2026-09-01 on the project's own page, narrows\nthis to 480 instances chosen for reproducible evaluation: instances with flaky or ungradeable tests were\ndropped, Docker environments were rebuilt to address dependency and browser drift, and JavaScript grading\nand visual test-asset handling were made more robust.\n\n## Who publishes it\n\nSWE-bench Multimodal comes from John Yang and Carlos E. Jimenez with Alex L. Zhang, Kilian Lieret, Joyce\nYang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik Narasimhan, Diyi Yang, Sida Wang\nand Ofir Press, submitted to arXiv in October 2024 and accepted to ICLR 2025. It is maintained today\nalongside the rest of the family on swebench.com, which issued the v2 dataset refresh in September 2026.\n\n## Lineage\n\nMultimodal is a sibling extension of the base SWE-bench (`swe_bench`) methodology, applying the same\ncollection and grading approach to a visual, JavaScript/TypeScript task set rather than to Python. It sits\nalongside SWE-bench Verified (`swe_bench_verified`), SWE-bench Multilingual (`swe_bench_multilingual`) and\nSWE-bench Pro as siblings rather than descendants of one another; no successor to Multimodal itself was\nfound.\n\n## Saturation and contamination\n\nAt the original release, the best system tested (SWE-agent) resolved only 12% of instances, with the\nnext-best system at 6% \u2014 a wide-open benchmark at the time, but that figure is now nearly two years old\nand describes the superseded 517-instance set rather than the current 480-instance v2. Because v2 went\nlive only a week before this page was researched, no post-refresh leaderboard or current top score could\nbe confirmed here; saturation is accordingly unknown rather than assumed low. Contamination risk is high\nfor the usual structural reason (every instance is a real, dated, merged pull request), with the added and\nless-studied wrinkle that a training corpus could also have absorbed the screenshots and mockups\nthemselves.\n\n## How to run it\n\nMultimodal runs on the SWE-bench Docker evaluation harness and `swebench` CLI, extended to serve\nimage-bearing issues and to grade JavaScript/TypeScript test output. It was not found in the\nlm-evaluation-harness, HELM, OpenCompass or BIG-bench task lists. Because the harness must additionally\nhand a model any embedded images, comparing scores across publishers also means checking whether the\nsystem under test actually has vision input at all, not just whether it has repository and shell access.\n\n## Reading the numbers\n\nA high Multimodal score would show a system can act on visual information embedded in a real software\ntask, not only read it \u2014 a capability plain-text SWE-bench cannot probe at all. Given how low the only\npublished baseline is (12%) and how recent the v2 refresh is, a reader should treat any current Multimodal\nnumber as preliminary and check whether it was run against the original 517-instance set or the September\n2026 480-instance v2, since the two are not the same benchmark despite the shared name.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_multimodal",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Mythos Preview",
   "model_id": "anthropic/claude-mythos-preview",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 59.0,
   "source": "anthropic-system-card"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 27.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  }
 ],
 "page": {
  "aliases": [
   "SWE-bench M",
   "SWE-bench MM"
  ],
  "category": "coding",
  "contamination": {
   "note": "Same structural risk as the rest of the family: every instance is a real, publicly merged pull request, so a model trained after an instance's fix date may have seen it. Visual assets (screenshots, mockups) add a second, less-studied exposure path distinct from the family's usual text-only concern.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "JavaScript",
    "TypeScript"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "code",
    "text",
    "image"
   ],
   "public_test_set": true,
   "size": 480,
   "size_note": "The original 2024 release held 517 task instances containing at least one image in the issue text or its tests, drawn from JavaScript/TypeScript repositories. A \"v2\" refresh released 2026-09-01 retains 480 of those instances chosen for reproducible evaluation, dropping ones with flaky or ungradeable tests, rebuilding the Docker environments to fix dependency and browser drift, and hardening JavaScript grading and visual test-asset handling.\n",
   "splits": "single test split; 517 instances at original release, 480 in the v2 refresh",
   "url": "https://huggingface.co/datasets/SWE-bench/SWE-bench_Multimodal"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice M",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Uses the SWE-bench Docker evaluation harness and `swebench` CLI, extended for image-bearing issues and JavaScript/TypeScript test grading. Not confirmed in the lm-evaluation-harness, HELM, OpenCompass or BIG-bench task lists.\n"
  },
  "id": "swe_bench_multimodal",
  "last_updated": "2026-09",
  "leaderboard_url": "https://www.swebench.com/multimodal.html",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-bench Multimodal measures the same underlying skill as SWE-bench \u2014 resolving a real GitHub issue with a patch a project's own tests accept \u2014 but restricted to issues that include a visual element: a screenshot of a bug, a design mockup or wireframe, a diagram of the desired behaviour, or an error message with visual context. Repositories are JavaScript/TypeScript projects for web interface design, diagramming, data visualisation, syntax highlighting and interactive mapping, so a model must read and act on both text and images to succeed, not just text.\n",
  "metric": {
   "baseline_note": "No human baseline published. At the original 2024 release, SWE-agent resolved 12% of task instances, the best of the systems tested; the next-best system reached 6%.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved (FAIL_TO_PASS and PASS_TO_PASS tests both pass)",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench Multimodal",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2410.03859",
   "title": "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?",
   "url": "https://arxiv.org/abs/2410.03859",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "John Yang",
    "Carlos E. Jimenez",
    "Alex L. Zhang",
    "Kilian Lieret",
    "Joyce Yang",
    "Xindi Wu",
    "Ori Press",
    "Niklas Muennighoff",
    "Gabriel Synnaeve",
    "Karthik R. Narasimhan",
    "Diyi Yang",
    "Sida I. Wang",
    "Ofir Press"
   ],
   "org": "Originally Princeton NLP / Stanford, with the SWE-bench team; maintained today by the SWE-bench Team at swebench.com",
   "url": "https://www.swebench.com/multimodal.html"
  },
  "released": "2024-10",
  "repo_url": "https://github.com/SWE-bench/SWE-bench",
  "saturation": {
   "as_of": "",
   "note": "At the original October 2024 release, the best system tested (SWE-agent) resolved 12% of instances, with the next-best system at 6% \u2014 far from any ceiling, but that figure is now nearly two years old and describes the superseded 517-instance set. The v2 refresh (480 instances) went live 2026-09-01, one week before this page was researched; no post-refresh leaderboard or top score could be confirmed in this research pass, so current standing is not established.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?",
    "url": "https://arxiv.org/abs/2410.03859"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Multimodal overview",
    "url": "https://www.swebench.com/multimodal.html"
   }
  ],
  "status": "active",
  "subcategory": "GitHub issue resolution / patch generation with visual inputs",
  "summary": "SWE-bench Multimodal tests GitHub-issue patching on JavaScript/TypeScript repositories where the issue includes an image, such as a bug screenshot or design mockup.",
  "tags": [
   "coding",
   "multimodal",
   "agentic",
   "github-issues",
   "patch-generation",
   "docker",
   "javascript"
  ],
  "task_format": "Given an issue description that includes at least one image, plus repository access, the system outputs a patch. The patch is applied inside a container and graded against FAIL_TO_PASS and PASS_TO_PASS tests recovered from the pull request that originally fixed the issue, the same protocol as the rest of the SWE-bench family.\n"
 }
}