{
 "body": "\n## What it measures\n\nSWE-bench-java measures GitHub issue resolution in Java. The agent gets an issue\nand a pre-fix snapshot of a Maven-built project, then must produce a patch that\nthe project's tests accept. The public set is the verified slice of 91 tasks\nfrom six widely used repositories, including Jackson, Gson, Dubbo, and Jib.\n\nThe authors built it as a first non-Python SWE-bench, using Verified-style\nhuman screens so underspecified issues and unfair tests are reduced.\n\n## How it is scored\n\nResolved rate is the share of the 91 instances where all listed tests pass after\nthe patch. That is the same fail-to-pass plus pass-to-pass rule as SWE-bench.\nTable 2 of the paper reports only SWE-agent runs. DeepSeek-V2 reached 9.89%\n(9/91); GPT-4o reached 6.59%. The paper says the SWE-agent port was hasty and\ndid not configure a runtime for every issue, so those figures are a lower bound\non what a fully wired Java agent might score in 2024.\n\n## Dataset and licence\n\nThe Hugging Face card Daoguang/Multi-SWE-bench is Apache-2.0 and contains\n`swe-bench-java-verified.json` (91 rows). That hub name is easy to confuse with\nByteDance's later [Multi-SWE-bench](multi_swe_bench.md). Construction started\nfrom 19 Java repositories and 1,979 crawled issues, then compile checks, test\nfilters, and ten Java developers using OpenAI's Verified annotation rubric.\n\n## Who publishes it\n\nDaoguang Zan, Zhirong Huang, Ailun Yu, Shaoxin Lin, and co-authors at Huawei\nand collaborating organisations posted arXiv:2408.14354 on 2024-08-26. The\ndataset went up on 2024-08-24 and was last modified 2024-09-03. The 2024 paper\nand the Java Docker README still point to multi-swe-bench.github.io; that\ndomain now titles itself Multi-SWE-bench, which is also the name of the later\nmultilingual successor. Do not read the live homepage as a 2026 Java-91 board.\n\n## Lineage\n\nThis is the Java precursor in the [SWE-bench](swe_bench.md) family. It is not\n[SWE-bench Multilingual](swe_bench_multilingual.md), which samples nine\nlanguages including Java in a 300-task SWE-bench-team set. Overlapping authors\nlater released Multi-SWE-bench (1,632 instances, seven languages, 2025-04),\nwhich is the broader successor rather than a rename of these 91 tasks.\n\n## Saturation and contamination\n\n9.89% in August 2024 is not a ceiling; it is an early SWE-agent snapshot.\nWhether 2026 agents saturate the 91-set was not established here. Contamination\nrisk is high because gold patches are public pull requests.\n\n## How to run it\n\nUse the published Docker environment and the `java_verified` JSON. State the\nagent scaffold. Do not drop these 91 Java tasks into a Multi-SWE-bench or\nSWE-bench Multilingual average without saying so.\n\n## Reading the numbers\n\nA SWE-bench-java score is Java-only and 91 instances, heavily weighted toward\njackson-databind (49 tasks). It is not a multilingual result. If you need\nbroader Java-plus-other-language coverage, look at Multi-SWE-bench or\nSWE-bench Multilingual instead, and do not treat 2024 SWE-agent percentages as\ntoday's frontier.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_java",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-bench-java-verified",
   "SWE-bench Java"
  ],
  "category": "coding",
  "contamination": {
   "note": "Instances are public merged Java pull requests with gold patches in the dataset, the same structural leak as original SWE-bench.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "Java"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 91,
   "size_note": "Final verified split: 91 instances from 6 repositories, counted in swe-bench-java-verified.json (google/gson 5, FasterXML/jackson-core 23, FasterXML/jackson-databind 49, FasterXML/jackson-dataformat-xml 5, apache/dubbo 4, GoogleContainerTools/jib 5). Construction crawled 1,979 issue instances from 19 repositories, kept 308 that compiled, 137 with fail-to-pass and no pass-to-fail tests, then 91 after 10 Java developers applied SWE-bench Verified-style screens.\n",
   "splits": "java_verified (91 instances)",
   "url": "https://huggingface.co/datasets/Daoguang/Multi-SWE-bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-074 (Codex coordinated)",
   "reviewed": "2026-09-09",
   "reviewed_by": "Grok Build independent review, batch-074"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Docker evaluation environment at multi-swe-bench/multi-swe-bench-env. Paper runs used SWE-agent. The authors note they did not configure a runtime for every Java issue before those SWE-agent experiments.\n"
  },
  "id": "swe_bench_java",
  "last_updated": "2024-09",
  "leaderboard_url": "https://multi-swe-bench.github.io",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench",
   "successors": [
    "multi_swe_bench"
   ],
   "variants": [
    "swe_bench_multilingual"
   ]
  },
  "measures": "SWE-bench-java tests whether a model can resolve a real Java GitHub issue by patching the repository at the pre-fix commit. It is the Java analogue of SWE-bench Verified: issues are filtered so they compile, have fail-to-pass tests, and survive developer screening.\n",
  "metric": {
   "baseline_note": "No human baseline. Paper Table 2 reports SWE-agent results on all 91 verified instances.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "resolved rate",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench-java",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2408.14354",
   "title": "SWE-bench-java: A GitHub Issue Resolving Benchmark for Java",
   "url": "https://arxiv.org/abs/2408.14354",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Daoguang Zan",
    "Zhirong Huang",
    "Ailun Yu",
    "Shaoxin Lin",
    "Yifan Shi",
    "Wei Liu",
    "Dong Chen",
    "Zongshuai Qi",
    "Hao Yu",
    "Lei Yu",
    "Dezhi Ran",
    "Muhan Zeng",
    "Bo Shen",
    "Pan Bian",
    "Guangtai Liang",
    "Bei Guan",
    "Pengjie Huang",
    "Tao Xie",
    "Yongji Wang",
    "Qianxiang Wang"
   ],
   "org": "Huawei and collaborators",
   "url": "https://multi-swe-bench.github.io"
  },
  "released": "2024-08",
  "repo_url": "https://github.com/multi-swe-bench/multi-swe-bench-env",
  "saturation": {
   "as_of": "2024-08",
   "note": "Paper Table 2: SWE-agent + DeepSeek-V2-0628 resolved 9.89% (9/91). Other listed runs: DeepSeekCoder-V2 7.69%, GPT-4o 6.59%, GPT-4o-mini 1.10%, Doubao-pro-128k 1.10%. No later verified-split leaderboard figure was confirmed. These 2024 agent scores should not be read as a 2026 ceiling.\n",
   "status": "open",
   "top_score": 9.89
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench-java (arXiv:2408.14354v1)",
    "url": "https://arxiv.org/abs/2408.14354"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench-java HTML full text",
    "url": "https://arxiv.org/html/2408.14354v1"
   },
   {
    "accessed": "2026-09-08",
    "title": "Daoguang/Multi-SWE-bench (SWE-bench-java-verified) dataset card",
    "url": "https://huggingface.co/datasets/Daoguang/Multi-SWE-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "swe-bench-java-verified.json instance file",
    "url": "https://huggingface.co/datasets/Daoguang/Multi-SWE-bench/resolve/main/swe-bench-java-verified.json"
   },
   {
    "accessed": "2026-09-09",
    "title": "multi-swe-bench/multi-swe-bench-env Java evaluation README",
    "url": "https://github.com/multi-swe-bench/multi-swe-bench-env"
   }
  ],
  "status": "active",
  "subcategory": "Java GitHub issue resolution / patch generation",
  "summary": "A 91-instance, human-screened Java SWE-bench set from six popular repositories, scored by fail-to-pass unit tests.",
  "tags": [
   "coding",
   "agentic",
   "java",
   "github-issues",
   "patch-generation",
   "docker"
  ],
  "task_format": "The system sees the issue text and a Dockerized Maven/Java checkout, then emits a patch. Tests from the fixing pull request grade the result. The authors evaluated SWE-agent rather than a one-shot RAG baseline.\n"
 }
}