{
 "body": "\n## What it measures\n\nRE-Bench measures issue resolution across software ecosystems beyond Python. An agent receives an issue and a repository state, edits files, and must produce a patch that addresses the issue. The benchmark covers Java, TypeScript, JavaScript, Go, Rust, C, and C++.\n\nThe paper reports 82% of human attempts making non-zero progress and 24% matching or exceeding strong reference solutions.\n\n## How it is scored\n\nSuccess is determined by the benchmark's repository tests and issue-resolution protocol, summarized as environment score. Reproduce results with the same agent harness, patch policy, test commands, timeout and environment. The paper compares Agentless, SWE-agent and OpenHands, but this page does not infer a universal score from those model-specific experiments.\n\n## Dataset and licence\n\nThe benchmark contains issue-resolution tasks tied to multilingual repositories and their test environments. The opened paper and repository establish the benchmark and public data release but do not establish a single dataset licence or a stable train/dev/test split for the 1,632 benchmark instances. Confirm the repository and dataset cards before redistribution.\n\n## Who publishes it\n\nRE-Bench is maintained in the official `multi-swe-bench/multi-swe-bench` repository and described by the paper \u201cRE-Bench: A Multilingual Benchmark for Issue Resolving\u201d (arXiv:2411.15114, 2024). The opened sources did not provide a complete author list in a form suitable for this record.\n\n## Lineage\n\nRE-Bench extends the issue-resolution setting associated with SWE-bench to seven programming languages. The authors also describe Multi-SWE-RL as a related, larger release for training research, rather than a successor benchmark score set.\n\n## Saturation and contamination\n\nThe repositories and issue data are public, so models or agents exposed to them can be contaminated. The opened sources do not establish a contamination study, rotating holdout or current saturation ceiling.\n\n## How to run it\n\nUse the official repository's task environments and test commands. Record the language, repository revision, agent harness, patch policy, timeout, dependency setup and test outcomes for every task. Report aggregate environment score together with per-language results.\n\n## Reading the numbers\n\nA higher environment score means more issue instances were resolved under the selected execution protocol. Results depend on environment setup, tests, agent interaction budget and whether failures are caused by patch quality or infrastructure. Compare runs only when those conditions match.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "re_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "RE-Bench: A Multilingual Benchmark for Issue Resolving"
  ],
  "category": "agentic",
  "contamination": {
   "note": "The benchmark and repositories are public; the source does not establish a contamination audit or private rotating holdout.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 7,
   "size_note": "The paper reports seven environments and human data from 71 eight-hour attempts by 61 experts.",
   "splits": "The opened sources do not establish a stable train/dev/test split for the 1,632 benchmark instances.",
   "url": "https://huggingface.co/datasets/ByteDance-Seed/RE-Bench"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-c-005 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "The paper evaluates Agentless, SWE-agent and OpenHands; use the repository's task environments and test execution protocol."
  },
  "id": "re_bench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "AI research engineering evaluation",
   "predecessor": "SWE-bench",
   "successors": [],
   "variants": []
  },
  "measures": "Progress and research-engineering performance in realistic ML environments under specified time budgets.",
  "metric": {
   "baseline_note": "The paper evaluates issue-resolution success with representative agent methods; no random or human baseline is established here.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "environment score",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "RE-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2411.15114",
   "title": "RE-Bench: A Multilingual Benchmark for Issue Resolving",
   "url": "https://arxiv.org/abs/2411.15114",
   "year": 2024
  },
  "publisher": {
   "authors": [],
   "org": "RE-Bench authors",
   "url": "https://arxiv.org/abs/2411.15114v2"
  },
  "released": "2024-11",
  "repo_url": "https://arxiv.org/abs/2411.15114v2",
  "saturation": {
   "as_of": "",
   "note": "No current saturation ceiling was established in the sources opened for this page.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "RE-Bench paper",
    "url": "https://arxiv.org/abs/2411.15114"
   },
   {
    "accessed": "2026-09-09",
    "title": "Official RE-Bench repository",
    "url": "https://arxiv.org/abs/2411.15114v2"
   }
  ],
  "status": "active",
  "subcategory": "research engineering agents",
  "summary": "RE-Bench evaluates agents that modify repositories to resolve issues across Java, TypeScript, JavaScript, Go, Rust, C and C++.",
  "tags": [
   "agentic",
   "agents",
   "multilingual",
   "issue-resolution"
  ],
  "task_format": "Open-ended agent interaction with seven research-engineering environments and objective environment scores."
 }
}