SWE-bench Agent

An internal model-card key for an agentic (not single-shot patch) SWE-bench score; no publisher, paper or dataset for it under this name was found.

unassessed

What it measures

`swe_bench_agent` is a scoring key used by this repository's own model-card template, not a benchmark published anywhere under that name. The template comments it as "agentic SWE-bench (not just patch gen)," grouped with other agentic-environment benchmarks (tau_bench, web_arena, os_world) rather than with the SWE-bench family's other named variants. What exact dataset, instance count, harness or scaffold produces the number is not documented anywhere this research could find.

Task format

Not documented under this name. If the template comment is accurate, it denotes SWE-bench evaluated by an agent that reads, runs and edits a repository over multiple steps, as opposed to a single-shot patch generated from one prompt — but no source specific to `swe_bench_agent` describes its dataset, instance count or grading protocol.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub