SWE-Bench-Mutated is a tool and methodology that rewrites formal SWE-bench issue text into realistic chat-style developer queries, to test agents on more realistic inputs.
unassessed
| Category | coding |
|---|---|
| Subcategory | benchmark mutation / realistic query generation |
| Page status | active |
| Metric | % resolved (as inherited from the underlying benchmark being mutated) |
| Direction | higher_is_better |
| Unit | percent |
| Dataset licence | MIT |
| Publisher | Microsoft |
SWE-Bench-Mutated addresses a gap between how coding-agent benchmarks are written and how developers actually phrase requests to chat-based coding assistants. It uses an LLM to rewrite existing, formally-worded SWE-bench issue descriptions into realistic, informally-phrased user queries based on patterns observed in real developer interactions with chat-based agents, and packages the result as a dataset for downstream agent evaluation. The accompanying paper reports that formally-worded benchmarks can overestimate some agents' real-world capability by more than 50% relative to their performance on the mutated, realistic-query version.
A CLI tool takes an existing benchmark's task descriptions (SWE-bench Verified, the TypeScript portion of Multi-SWE-Bench, and an internal Microsoft C# benchmark were used in the paper) and uses an LLM to rewrite each into a realistic user-style query, saving the result as a new dataset that can be run through the same agent-evaluation pipeline as the source benchmark.
No model card in ModelSpec reports this benchmark yet.