{
 "body": "\n## What it measures\n\nSWE-bench Science asks a coding agent to repair or extend real scientific software.\nThe model gets a repository at a fixed commit and must produce a change that keeps\nscientific contracts intact: units, coordinate systems, file formats, and numerical\ninvariants. Ordinary GitHub-issue benchmarks rarely test those contracts.\n\nEach of the 119 tasks sits in one of three paradigms. Issue-driven tasks follow a\nreported defect. Expert-exploratory tasks need domain judgement. Engineering-\nintegration tasks stitch scientific code into a larger workflow. Most tasks are\nPython; a minority use C, C++, Fortran, or MATLAB/Octave.\n\n## How it is scored\n\nThe headline metric is pass@1 on private verifier tests. A trial counts only if\nevery applicable held-out test passes after a clean rebuild. The public leaderboard\nalso lists public-test rate, fail-to-pass, pass-to-pass, and scores for the three\nparadigms. Public tests are much easier than private ones: the leading Claude Code\nrun is at 96.64% public and 47.90% overall. There is no published human baseline.\n\n## Dataset and licence\n\nThe release metadata and tooling are MIT. Upstream scientific sources keep their\nown licences; 23 tasks need `--allow-restricted-licenses` because of GPL-family\ncode or restricted materials (18 GPL-family rows plus five other restricted rows).\nHugging Face ships `data/tasks.csv` (119 rows, 98 distinct `repository_url` values)\nand thin Harbor bundles. Fine-grained `domain` strings in the CSV are more\nnumerous than the paper's 20 benchmark-level domains. Docker Hub holds one\nenvironment image and one verifier image per task, pinned by linux/amd64 digest.\nGold patches and private tests are not in the public dataset.\n\n## Who publishes it\n\nOpenMOSS authors Zhipeng Xu, Jiahao Lu, Yining Zheng, Yuxin Wang, and Xipeng Qiu\nposted arXiv:2608.19799 on 2026-08-20 (v2 on 2026-09-01). The dataset, GitHub\ndocs, and leaderboard are dated August\u2013September 2026. The project site is\nswescience.github.io.\n\n## Lineage\n\nThe work sits in the [SWE-bench](swe_bench.md) family as a scientific-computing\nvariant, not a drop-in replacement for [SWE-bench Verified](swe_bench_verified.md).\nIt borrows the repository-level, test-graded shape but uses Harbor/Pier instead of\nthe SWE-bench CLI, and it holds out verifier tests. It is unrelated to Lab-bench\nor other science QA sets.\n\n## Saturation and contamination\n\nTop overall pass@1 is 47.90% as of 2026-09-01, with a wide spread down to 7.56%.\nHard70 remains at 21.43% for the leader. The benchmark is open. Public scientific\nrepos create medium contamination risk; withheld verifiers reduce answer copying\nbut not exposure to the surrounding code.\n\n## How to run it\n\nDownload the Hugging Face dataset, materialize a selection with\n`scripts/materialize.py`, and run Pier (`datacurve-pier==0.3.0`) with Docker\nlinux/amd64 support. Name the selection (default 96, all 119, ablation 91, or\nHard70), the agent, and the model. Do not compare a public-test number with\noverall pass@1.\n\n## Reading the numbers\n\nA high overall pass@1 means the agent preserved private scientific checks, not\nthat it merely compiled or passed the public smoke tests. Report the paradigm\nsplit: engineering-integration is much harder than expert-exploratory for the\ncurrent leader (27.78% vs 65.31%). Pair the score with Verified or SWE-bench Pro\nif you need a general software-engineering baseline rather than a science-stack\none.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_science",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-bench-Science"
  ],
  "category": "coding",
  "contamination": {
   "note": "Source repositories are public scientific GitHub projects, so pretraining may include surrounding code. The release withholds gold patches, private verifier tests, credentials, and trajectories. That limits answer leakage relative to vanilla SWE-bench, but does not hide the public codebases.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "Python",
    "C++",
    "C",
    "Fortran",
    "MATLAB"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": false,
   "size": 119,
   "size_note": "119 release tasks (ids 001-119) from 98 GitHub repositories and 20 benchmark-level scientific domains. Default selection: 96 unrestricted tasks. Restricted selection: 23 tasks (18 GPL-family plus five others). A 91-task science-knowledge ablation split is marked in tasks.csv. A Hard70 subset was defined on 2026-08-28 from the 70 tasks with lowest mean reward across 12 complete model runs. Per-task language labels are mostly Python (103), with smaller C++, C, Fortran, MATLAB/Octave, and mixed bindings counts.\n",
   "splits": "119 tasks; default 96 unrestricted; 23 restricted opt-in; 91-task ablation; Hard70 subset",
   "url": "https://huggingface.co/datasets/OpenMOSS-Team/SWE-bench-Science"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-074 (Codex coordinated)",
   "reviewed": "2026-09-09",
   "reviewed_by": "Grok Build independent review, batch-074"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Harbor-format tasks run with Pier (pinned datacurve-pier==0.3.0). Agents such as Claude Code, Codex, and mini-swe-agent are selected at runtime. Not confirmed in lm-evaluation-harness, HELM, OpenCompass, or BIG-bench.\n"
  },
  "id": "swe_bench_science",
  "last_updated": "2026-09",
  "leaderboard_url": "https://swescience.github.io/",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench",
   "successors": [],
   "variants": [
    "swe_bench_verified"
   ]
  },
  "measures": "SWE-bench Science tests whether a coding agent can change a real scientific computing repository while preserving domain contracts such as units, file formats, numerics, and geometry. Each task starts from a fixed baseline commit and is checked in a clean environment, not by matching a gold patch.\n",
  "metric": {
   "baseline_note": "Pass@1 requires every applicable private test to pass. The leaderboard also reports public/private, fail-to-pass, pass-to-pass, and per-paradigm rates. No human baseline is published.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench Science",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2608.19799",
   "title": "SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?",
   "url": "https://arxiv.org/abs/2608.19799",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Zhipeng Xu",
    "Jiahao Lu",
    "Yining Zheng",
    "Yuxin Wang",
    "Xipeng Qiu"
   ],
   "org": "OpenMOSS",
   "url": "https://swescience.github.io/"
  },
  "released": "2026-08",
  "repo_url": "https://github.com/OpenMOSS/SWE-bench-Science",
  "saturation": {
   "as_of": "2026-09",
   "note": "Leaderboard updated 2026-09-01: Claude-Opus-5 (max) with Claude Code at 47.90% overall pass@1, then DeepSeek-V4-Pro (max) 42.02% and GPT-5.6-sol (max) 40.34%. Hard70 is much lower (21.43% for the same Claude run). Public tests are near ceiling for top models; private tests are the binding constraint.\n",
   "status": "open",
   "top_score": 47.9
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Science (arXiv:2608.19799v2)",
    "url": "https://arxiv.org/abs/2608.19799"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenMOSS-Team/SWE-bench-Science dataset card",
    "url": "https://huggingface.co/datasets/OpenMOSS-Team/SWE-bench-Science"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenMOSS/SWE-bench-Science repository",
    "url": "https://github.com/OpenMOSS/SWE-bench-Science"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Science leaderboard",
    "url": "https://swescience.github.io/"
   },
   {
    "accessed": "2026-09-09",
    "title": "SWE-bench Science release statistics",
    "url": "https://huggingface.co/datasets/OpenMOSS-Team/SWE-bench-Science/raw/main/data/statistics.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "SWE-bench Science tasks.csv",
    "url": "https://huggingface.co/datasets/OpenMOSS-Team/SWE-bench-Science/raw/main/data/tasks.csv"
   },
   {
    "accessed": "2026-09-09",
    "title": "SWE-bench Science MIT licence",
    "url": "https://raw.githubusercontent.com/OpenMOSS/SWE-bench-Science/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "scientific software engineering / repository-level repair",
  "summary": "119 repository-level scientific coding tasks across 98 GitHub projects and 20 domains, scored by held-out programmatic verifiers.",
  "tags": [
   "coding",
   "agentic",
   "scientific-computing",
   "docker",
   "harbor",
   "pier",
   "long-horizon"
  ],
  "task_format": "The agent works in a pinned environment image, then a separate verifier image applies the candidate patch, rebuilds if needed, and runs held-out tests. Tasks are grouped into issue-driven, expert-exploratory, and engineering-integration paradigms. Default runs use 96 unrestricted-license tasks; 23 restricted-license tasks require an explicit opt-in.\n"
 }
}