{
 "body": "\n## What it measures\n\nSWE-Bench ProMax measures large-scale refactoring, not bug fixing. The agent\nreceives a rewritten specification and a repository from before a real refactor\ncommit. It must coordinate edits across many files while keeping behaviour that\nthe test suite encodes. Gold patches average 11.4 source files and 261.6 lines,\nfar above typical SWE-bench Verified edits.\n\nThe set covers seven languages: Python, Java, TypeScript, Go, C, C++, and Rust.\nTasks with thin cross-file scope were dropped during curation.\n\n## How it is scored\n\nResolve rate is pass@1: the final tree must pass every test. The paper evaluates\nall 170 instances and also reports per-language rates, dollars per instance, and\nagent steps. Scaffold choice is part of the result. GPT-5.2 moves from 21.8%\nunder mini-swe-agent to 41.2% under OpenHands. There is no human baseline.\n\n## Dataset and licence\n\nThe public release is 170 JSON instances plus `eval.json` scripts, on Hugging\nFace as swe-bench-promax/SWE-Bench-ProMax (created 2026-08-10; last modified\n2026-08-26). Source repositories use ordinary open-source licences such as MIT,\nApache-2.0, BSD, and some GPL/AGPL; the dataset card itself does not declare a\nlicence, so that field is left empty. Gold patches and tests are in the\ndownload. Issue descriptions were rewritten so they do not quote the gold diff.\n\n## Who publishes it\n\nYuling Shi, Jinghan Xu, Kelin Fu, Wenhao Zeng, Shilin He, Xiaodong Gu, Shing-Chi\nCheung, and co-authors submitted the paper on 2026-08-10. The dataset card says\nit was accepted to COLM 2026. Contact listed on the card is yuling.shi@sjtu.edu.cn.\nNo separate GitHub organisation URL resolved during this research.\n\n## Lineage\n\nProMax is a SWE-bench-family stress test aimed at Verified saturation and at\nthin refactoring sets such as RefactorBench. It is not [SWE-bench Pro](swe_bench_pro.md)\nfrom Scale AI, despite the similar name. SWE-bench Pro targets long-horizon\nissue resolution with copyleft and private code. ProMax targets multilingual\nrefactoring of post-2025 public commits. A promised v2 from post-2026 issues\nis not released.\n\n## Saturation and contamination\n\n41.2% resolved is well below Verified's 75%+ regime, and language scores vary\nwidely, so the set is open. Contamination risk is medium by construction:\ncommits are after January 2025 and prompts are rewritten, but the code is still\npublic GitHub.\n\n## How to run it\n\nUse the dataset's Docker runner with a predictions file of `instance_id` and\n`model_patch` pairs. Record scaffold, model, and cost. Do not mix OpenHands\nnumbers with mini-swe-agent numbers. No lm-eval task name was found.\n\n## Reading the numbers\n\nA 40% ProMax score means the agent can finish some multi-file refactors, not\nthat it matches Verified bug-fix skill. Failed runs in the paper often edit\nfewer files than the gold patch and burn extra steps. Compare per-language\nrates before treating an overall figure as multilingual competence. TypeScript\nis concentrated: the paper says 28 TypeScript instances come from two\nrepositories, with Angular contributing 25.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_promax",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-Bench-ProMax",
   "SWE-bench ProMax"
  ],
  "category": "coding",
  "contamination": {
   "note": "Commits are public but dated after January 2025, and issue text is rewritten so the gold patch is not sitting in the prompt. That is weaker protection than a private corpus, and stronger than shipping raw commit messages.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "Python",
    "Java",
    "TypeScript",
    "Go",
    "C",
    "C++",
    "Rust"
   ],
   "license": "",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 170,
   "size_note": "170 instances from 70 repositories after filtering 29,782 candidates. Language counts on the dataset card: Python 29, TypeScript 28, Java 26, Go 23, C++ 22, Rust 22, C 20. Gold patches average 11.4 source files and 261.6 lines of code. Test patches add about 4.5 files and 185.5 lines.\n",
   "splits": "single test split of 170 instances",
   "url": "https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-074 (Codex coordinated)",
   "reviewed": "2026-09-09",
   "reviewed_by": "Grok Build independent review, batch-074"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Dataset-card runner: python src/evaluation/test_run.py with Docker, a preds.json file, swe-bench-promax.json, and eval.json. Paper results use OpenHands and mini-swe-agent. Not confirmed in lm-eval, HELM, OpenCompass, or BIG-bench.\n"
  },
  "id": "swe_bench_promax",
  "last_updated": "2026-08",
  "leaderboard_url": "",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench_verified",
   "successors": [],
   "variants": [
    "swe_bench_pro"
   ]
  },
  "measures": "SWE-Bench ProMax tests whether an agent can carry out a large, behaviour-preserving refactor in a real repository. Instances come from post-2025 GitHub commits tagged as refactoring, not from bug-fix issues. The agent must change many files so that a reviewed test suite still passes.\n",
  "metric": {
   "baseline_note": "No human baseline is published. The paper also reports API cost and step counts per instance.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "resolve rate (pass@1)",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-Bench ProMax",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2608.09802",
   "title": "SWE-Bench ProMax: Benchmarking Agents on Large-Scale Multilingual Code Refactoring",
   "url": "https://arxiv.org/abs/2608.09802",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Yuling Shi",
    "Jinghan Xu",
    "Kelin Fu",
    "Wenhao Zeng",
    "Shilin He",
    "Lei Zhang",
    "Yue Liu",
    "Zelin Zhao",
    "Terry Yue Zhuo",
    "Jialun Cao",
    "Siyu Ye",
    "Tianyu Liu",
    "Kai Cai",
    "Shing-Chi Cheung",
    "Xiaodong Gu"
   ],
   "org": "SWE-Bench-ProMax authors",
   "url": "https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax"
  },
  "released": "2026-08",
  "repo_url": "",
  "saturation": {
   "as_of": "2026-08",
   "note": "In the COLM 2026 paper, GPT-5.2 with OpenHands resolved 41.2% of 170 instances. Claude Sonnet 4.6 reached 38.8% on the same scaffold. mini-swe-agent scores are much lower (21.8% for GPT-5.2). No independent public leaderboard was found.\n",
   "status": "open",
   "top_score": 41.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-Bench ProMax (arXiv:2608.09802v1)",
    "url": "https://arxiv.org/abs/2608.09802"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-Bench ProMax HTML full text",
    "url": "https://arxiv.org/html/2608.09802v1"
   },
   {
    "accessed": "2026-09-08",
    "title": "swe-bench-promax/SWE-Bench-ProMax dataset card",
    "url": "https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax"
   }
  ],
  "status": "active",
  "subcategory": "multilingual repository-level refactoring",
  "summary": "170 expert-curated multilingual refactoring tasks from post-2025 commits; gold patches average 11.4 files and 261.6 lines.",
  "tags": [
   "coding",
   "agentic",
   "refactoring",
   "multilingual",
   "docker",
   "patch-generation"
  ],
  "task_format": "Given a rewritten issue description and a Dockerized pre-refactor checkout, the agent edits the tree. An instance is resolved only if every test in the suite passes. Issue text is rewritten from scratch so commit messages do not leak the gold patch.\n"
 }
}