{
 "body": "\n## What it measures\n\nMulti-SWE-bench measures issue resolution across software ecosystems beyond Python. An agent receives an issue and a repository state, edits files, and must produce a patch that addresses the issue. The benchmark covers Java, TypeScript, JavaScript, Go, Rust, C, and C++.\n\nThe paper reports 1,632 high-quality instances selected from 2,456 candidates and annotated by 68 expert annotators. It also describes a separate 4,723-instance Multi-SWE-RL release for reinforcement-learning research; that larger collection should not be confused with the benchmark count.\n\n## How it is scored\n\nSuccess is determined by the benchmark's repository tests and issue-resolution protocol, summarized as pass rate. Reproduce results with the same agent harness, patch policy, test commands, timeout and environment. The paper compares Agentless, SWE-agent and OpenHands, but this page does not infer a universal score from those model-specific experiments.\n\n## Dataset and licence\n\nThe benchmark contains issue-resolution tasks tied to multilingual repositories and their test environments. The opened paper and repository establish the benchmark and public data release but do not establish a single dataset licence or a stable train/dev/test split for the 1,632 benchmark instances. Confirm the repository and dataset cards before redistribution.\n\n## Who publishes it\n\nMulti-SWE-bench is maintained in the official `multi-swe-bench/multi-swe-bench` repository and described by the paper \u201cMulti-SWE-bench: A Multilingual Benchmark for Issue Resolving\u201d (arXiv:2504.02605, 2025). The opened sources did not provide a complete author list in a form suitable for this record.\n\n## Lineage\n\nMulti-SWE-bench extends the issue-resolution setting associated with SWE-bench to seven programming languages. The authors also describe Multi-SWE-RL as a related, larger release for training research, rather than a successor benchmark score set.\n\n## Saturation and contamination\n\nThe repositories and issue data are public, so models or agents exposed to them can be contaminated. The opened sources do not establish a contamination study, rotating holdout or current saturation ceiling.\n\n## How to run it\n\nUse the official repository's task environments and test commands. Record the language, repository revision, agent harness, patch policy, timeout, dependency setup and test outcomes for every task. Report aggregate pass rate together with per-language results.\n\n## Reading the numbers\n\nA higher pass rate means more issue instances were resolved under the selected execution protocol. Results depend on environment setup, tests, agent interaction budget and whether failures are caused by patch quality or infrastructure. Compare runs only when those conditions match.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "multi_swe_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving"
  ],
  "category": "coding",
  "contamination": {
   "note": "The benchmark and repositories are public; the source does not establish a contamination audit or private rotating holdout.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "java",
    "typescript",
    "javascript",
    "go",
    "rust",
    "c",
    "cpp"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 1632,
   "size_note": "The paper reports 1,632 high-quality benchmark instances annotated from 2,456 candidates by 68 expert annotators; it separately reports 4,723 Multi-SWE-RL instances.",
   "splits": "The opened sources do not establish a stable train/dev/test split for the 1,632 benchmark instances.",
   "url": "https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-c-005 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The paper evaluates Agentless, SWE-agent and OpenHands; use the repository's task environments and test execution protocol."
  },
  "id": "multi_swe_bench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "software-engineering issue resolution",
   "predecessor": "SWE-bench",
   "successors": [],
   "variants": []
  },
  "measures": "Whether an agent can produce a patch that resolves a real issue and passes the repository's tests.",
  "metric": {
   "baseline_note": "The paper evaluates issue-resolution success with representative agent methods; no random or human baseline is established here.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass rate",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "Multi-SWE-bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2504.02605",
   "title": "Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving",
   "url": "https://arxiv.org/abs/2504.02605",
   "year": 2025
  },
  "publisher": {
   "authors": [],
   "org": "Multi-SWE-bench authors",
   "url": "https://github.com/multi-swe-bench/multi-swe-bench"
  },
  "released": "2025-04",
  "repo_url": "https://github.com/multi-swe-bench/multi-swe-bench",
  "saturation": {
   "as_of": "",
   "note": "No current saturation ceiling was established in the sources opened for this page.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "Multi-SWE-bench paper",
    "url": "https://arxiv.org/abs/2504.02605"
   },
   {
    "accessed": "2026-09-09",
    "title": "Official Multi-SWE-bench repository",
    "url": "https://github.com/multi-swe-bench/multi-swe-bench"
   }
  ],
  "status": "active",
  "subcategory": "multilingual issue resolution",
  "summary": "Multi-SWE-bench evaluates agents that modify repositories to resolve issues across Java, TypeScript, JavaScript, Go, Rust, C and C++.",
  "tags": [
   "coding",
   "agents",
   "multilingual",
   "issue-resolution"
  ],
  "task_format": "Issue statement, repository snapshot and test environment; the agent edits the repository and is evaluated by tests."
 }
}