{
 "body": "## What it measures\n\nSWE-Bench-Mutated is not a fixed task set but a mutation methodology and accompanying CLI tool: it\nuses an LLM to rewrite an existing benchmark's formally-worded issue descriptions into realistic,\ninformally-phrased queries of the kind a developer would actually type into a chat-based coding\nassistant. The goal is to test whether an agent's measured capability on a formal benchmark still\nholds when the same underlying task is presented the way it would arise in practice.\n\nThe associated paper, \"Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent\nEvaluation,\" argues that formally-worded benchmarks like the original SWE-bench issue text can\noverestimate some agents' real-world capability, reporting a gap of more than 50% relative to\nperformance on the mutated version for some models on public benchmarks.\n\n## How it is scored\n\nMutation does not change how a task is graded: the resulting dataset is evaluated with the resolution\ncriteria of whichever base benchmark supplied the original task (for example, SWE-bench Verified's\nFAIL_TO_PASS/PASS_TO_PASS test protocol). What changes is the phrasing of the input the agent receives,\nso a drop in resolved rate after mutation is interpreted as an input-robustness gap rather than a\nchange in task difficulty at the code level.\n\n## Dataset and licence\n\nThere is no single fixed SWE-Bench-Mutated dataset with a published size: the repository is a tool\nthat generates a mutated dataset from a chosen source benchmark. The paper applies it to SWE-Bench\nVerified (500 instances), the TypeScript portion of Multi-SWE-Bench, and an internal, unreleased C#\nbenchmark. The tool's code is released under the MIT licence; the licence of any specific generated\ndataset would follow the licence of the base benchmark it was derived from, which is not established\nhere.\n\n## Who publishes it\n\nThe tool and paper come from Microsoft, authored by Spandan Garg, Benjamin Steenhoek, and Yufan Huang.\nThe paper is accepted at CAIN 2026 (the International Conference on AI Engineering, Research Track).\nNo separate public leaderboard was found in the sources reviewed.\n\n## Lineage\n\nSWE-Bench-Mutated is a derivative methodology applied to SWE-Bench Verified in the paper's public\nexperiments, so it is recorded in the SWE-bench family with SWE-Bench Verified as the benchmark it\nmutates. It is also applied to Multi-SWE-Bench (TypeScript) and an internal C# benchmark, neither of\nwhich have a page in this repository at the time of writing. No successor to this mutation approach\nwas found.\n\n## Saturation and contamination\n\nSaturation status is unknown in the conventional sense, since the tool does not produce one fixed,\nscored dataset; the paper's headline result is a drop in measured resolution rate for some models when\nmoving from formal to mutated queries on the same underlying tasks, which is a robustness finding\nrather than a saturation claim. Contamination risk is unknown: mutated query text is freshly generated\nby an LLM and less likely to be memorized verbatim, but the underlying issues and repository code\nstill come from public benchmarks, and no contamination study was found in the sources reviewed.\n\n## How to run it\n\nThe reference implementation is the CLI in the microsoft/SWE-Bench-Mutated-CAIN26 GitHub repository,\nwhich by default uses Azure OpenAI (via LiteLLM) to perform the rewriting, with alternative LLM\nproviders and mutation prompts configurable. There is no lm-evaluation-harness, HELM, or OpenCompass\ntask name, since this is a preprocessing tool rather than a packaged evaluation task; once a mutated\ndataset is generated, it is evaluated with whatever harness scores the base benchmark it came from.\n\n## Reading the numbers\n\nThere is no single \"SWE-Bench-Mutated score\" to compare across models in the way there is for\nSWE-bench Verified, because the mutated dataset depends on which base benchmark and which mutation\nmodel and prompts were used to generate it. What is comparable is the gap between an agent's score on\na formal benchmark and its score on that benchmark's mutated version: a large gap suggests the agent's\nmeasured capability is sensitive to how literally and formally a task is phrased, which the paper\nargues is a more realistic stress test than the original phrasing alone. Because the mutation itself\nuses an LLM, results can also vary with the mutation model and prompt configuration, so exact figures\nshould always be read alongside which base benchmark, mutation model, and prompt set produced them.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_mutated",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "Because the mutated text is newly generated by an LLM rather than scraped, direct memorization of the mutated phrasing is less likely, but the underlying issues and code are drawn from public benchmarks; no contamination study was found in the sources reviewed.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "Python",
    "TypeScript",
    "C#"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "No fixed dataset size: the tool generates a mutated dataset from whichever base benchmark it is applied to; the paper applies it to SWE-Bench Verified (500 instances), the TypeScript subset of Multi-SWE-Bench, and an internal C# benchmark not released publicly.",
   "splits": "",
   "url": "https://github.com/microsoft/SWE-Bench-Mutated-CAIN26"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Standalone CLI tool in the linked repository; evaluation of the resulting dataset uses whichever harness evaluates the base benchmark being mutated."
  },
  "id": "swe_bench_mutated",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench_verified",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-Bench-Mutated addresses a gap between how coding-agent benchmarks are written and how developers actually phrase requests to chat-based coding assistants. It uses an LLM to rewrite existing, formally-worded SWE-bench issue descriptions into realistic, informally-phrased user queries based on patterns observed in real developer interactions with chat-based agents, and packages the result as a dataset for downstream agent evaluation. The accompanying paper reports that formally-worded benchmarks can overestimate some agents' real-world capability by more than 50% relative to their performance on the mutated, realistic-query version.\n",
  "metric": {
   "baseline_note": "The tool does not define a new metric; it changes the task's input phrasing and evaluation continues to use the resolution criteria of whichever base benchmark was mutated.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved (as inherited from the underlying benchmark being mutated)",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "SWE-Bench-Mutated",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2510.08996",
   "title": "Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation",
   "url": "https://arxiv.org/abs/2510.08996",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Spandan Garg",
    "Benjamin Steenhoek",
    "Yufan Huang"
   ],
   "org": "Microsoft",
   "url": "https://github.com/microsoft/SWE-Bench-Mutated-CAIN26"
  },
  "released": "",
  "repo_url": "https://github.com/microsoft/SWE-Bench-Mutated-CAIN26",
  "saturation": {
   "as_of": "",
   "note": "The tool does not itself have a fixed leaderboard; the paper's headline finding is a drop in measured resolution rate for some models on mutated versus original queries, not a saturation claim.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation",
    "url": "https://arxiv.org/abs/2510.08996"
   },
   {
    "accessed": "2026-09-08",
    "title": "microsoft/SWE-Bench-Mutated-CAIN26 repository",
    "url": "https://github.com/microsoft/SWE-Bench-Mutated-CAIN26"
   }
  ],
  "status": "active",
  "subcategory": "benchmark mutation / realistic query generation",
  "summary": "SWE-Bench-Mutated is a tool and methodology that rewrites formal SWE-bench issue text into realistic chat-style developer queries, to test agents on more realistic inputs.",
  "tags": [
   "benchmark",
   "coding",
   "agentic",
   "evaluation-methodology",
   "robustness"
  ],
  "task_format": "A CLI tool takes an existing benchmark's task descriptions (SWE-bench Verified, the TypeScript portion of Multi-SWE-Bench, and an internal Microsoft C# benchmark were used in the paper) and uses an LLM to rewrite each into a realistic user-style query, saving the result as a new dataset that can be run through the same agent-evaluation pipeline as the source benchmark.\n"
 }
}