{
 "body": "\n## What it measures\n\nSWE-Bench-CL measures continual learning on software issues, not single-shot\nrepair. An agent walks a chronological sequence of Verified tasks from one\nPython repository, then another. The intended skills are accumulating project\nknowledge, transferring across related files, and keeping earlier fixes working.\n\nThe issues themselves are the familiar SWE-bench Verified bugs. What changes is\norder and scoring: tasks are grouped into eight repository curricula with\ndifficulty ramps, and file-overlap metadata marks possible dependencies.\n\n## How it is scored\n\nThe paper defines a performance matrix after each new task. From that matrix it\nderives average accuracy, forgetting, forward and backward transfer, tool-use\nefficiency, CL-plasticity, CL-stability, and a composite CL-Score. A single\n\"% resolved\" number is not the intended headline. No completed frontier-model\ntable on those metrics appears in the paper. A stock-harness trial is presented\nmainly to show that i.i.d. SWE-bench containers are a poor fit.\n\n## Dataset and licence\n\nThe GitHub repository is MIT-licensed. The JSON curriculum contains 273 tasks\nacross django, sympy, sphinx, matplotlib, scikit-learn, astropy, xarray, and\npytest. Construction keeps Verified fields (problem statement, gold patch, tests)\nand adds sequence position, difficulty, and overlapping files. There is no\nheld-out private split.\n\n## Who publishes it\n\nThomas Joshi, Shayan Chowdhury, and Fatih Uysal posted arXiv:2507.00014 on\n2025-06-13 (the 2507 identifier is an arXiv cs.LG overflow id). The README\nframes the work as a Columbia COMS 4995 course project with Prof. Richard Zemel,\nadvised by Tom Zollo. Code and data live at github.com/thomasjoshi/agents-never-forget.\nNo project leaderboard site was found. The paper outlines CL experiments as\nongoing rather than reporting a completed frontier-model table.\n\n## Lineage\n\nThis is a protocol on [SWE-bench Verified](swe_bench_verified.md), not a new\nissue pool. It should not be merged with Verified scores or with\n[SWE-bench Lite](swe_bench_lite.md). It is also not a live-updating set like\n[SWE-bench-Live](swe_bench_live.md). The authors note that SWE-bench dump\nimages target original SWE-bench and Lite, which is why they shipped their own\neval scripts.\n\n## Saturation and contamination\n\nSaturation is unknown; there is no public ranking of current agents on the CL\nmetrics. Contamination risk is high for the same reason Verified is high: gold\npatches are public. Sequential scoring can still reveal forgetting even when\nsome individual bugs are familiar.\n\n## How to run it\n\nUse the repository's `data/SWE-Bench-CL-Curriculum.json` and one of the\n`eval_v1`, `eval_v2_agent`, or `eval_v3_swe-agent` entry points. Say whether\nmemory retrieval was on, and whether you re-tested prior tasks after each new\nissue. Do not report a number from the official `swebench` CLI on these\nsequences as a SWE-Bench-CL score.\n\n## Reading the numbers\n\nA CL-Score or forgetting figure is only comparable under the same retest\nschedule and memory policy. A high final accuracy with large forgetting means\nthe agent is still solving isolated Verified bugs, not learning the repo.\nUntil a maintained leaderboard exists, treat published figures as protocol\ndemonstrations. Pair any result with a standard Verified score on the same\nmodel so the i.i.d. baseline is visible.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_cl",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-bench-CL"
  ],
  "category": "coding",
  "contamination": {
   "note": "Every item is a SWE-bench Verified instance, hence a public merged pull request. Sequencing does not hide gold patches. Continual-learning metrics can still be informative if the agent is not allowed to reread hidden tests.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "Python"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 273,
   "size_note": "Eight sequences from SWE-bench Verified repositories with at least 15 tasks each: django/django 50, sympy/sympy 50, sphinx-doc/sphinx 44, matplotlib/matplotlib 34, scikit-learn/scikit-learn 32, astropy/astropy 22, pydata/xarray 22, pytest-dev/pytest 19 (273 total). Difficulty buckets follow Verified wall-clock labels (<15 min, 15 min-1 h, 1-4 h, >4 h).\n",
   "splits": "eight repository sequences; no separate train/test split",
   "url": "https://github.com/thomasjoshi/agents-never-forget"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-074 (Codex coordinated)",
   "reviewed": "2026-09-09",
   "reviewed_by": "Grok Build independent review, batch-074"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Reference code is eval_v1 (SWE-bench Docker harness), eval_v2_agent (LangGraph plus FAISS), and eval_v3_swe-agent in thomasjoshi/agents-never-forget. The README also names eval_v3_agent; the repository folder is eval_v3_swe-agent. The authors report that official SWE-bench dump containers do not line up cleanly with this Verified-derived order.\n"
  },
  "id": "swe_bench_cl",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench_verified",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-Bench-CL asks whether a coding agent improves, transfers, and avoids forgetting as it walks a stream of GitHub issues from one repository. The underlying issues are SWE-bench Verified tasks, reordered into curricula instead of scored as i.i.d. bugs.\n",
  "metric": {
   "baseline_note": "The paper defines ACC, forgetting, forward/backward transfer, tool-use efficiency, CL-P/CL-S, and a composite CL-Score. It does not publish a completed frontier-model table on those metrics.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "average accuracy (plus forgetting, transfer, and CL-Score)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "SWE-Bench-CL",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2507.00014",
   "title": "SWE-Bench-CL: Continual Learning for Coding Agents",
   "url": "https://arxiv.org/abs/2507.00014",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Thomas Joshi",
    "Shayan Chowdhury",
    "Fatih Uysal"
   ],
   "org": "Independent course project (COMS 4995)",
   "url": "https://github.com/thomasjoshi/agents-never-forget"
  },
  "released": "2025-06",
  "repo_url": "https://github.com/thomasjoshi/agents-never-forget",
  "saturation": {
   "as_of": "",
   "note": "No maintained leaderboard was found. A trial against the stock SWE-bench harness produced low pass rates that the authors attribute to container mismatch with Verified-derived sequences, not to a calibrated CL ranking.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-Bench-CL (arXiv:2507.00014v1)",
    "url": "https://arxiv.org/abs/2507.00014"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-Bench-CL HTML full text",
    "url": "https://arxiv.org/html/2507.00014v1"
   },
   {
    "accessed": "2026-09-08",
    "title": "thomasjoshi/agents-never-forget README",
    "url": "https://github.com/thomasjoshi/agents-never-forget"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-Bench-CL MIT licence",
    "url": "https://raw.githubusercontent.com/thomasjoshi/agents-never-forget/main/LICENSE"
   }
  ],
  "status": "proposed",
  "subcategory": "continual learning over SWE-bench Verified sequences",
  "summary": "A 273-task continual-learning reformulation of SWE-bench Verified: eight chronological repository sequences with forgetting and transfer metrics.",
  "tags": [
   "coding",
   "agentic",
   "continual-learning",
   "github-issues",
   "python"
  ],
  "task_format": "Eight repository sequences (273 tasks total) are ordered first by issue creation time, then by human-estimated fix time. After each task, evaluation can re-test earlier tasks. The authors also describe a LangGraph agent with FAISS memory as a reference scaffold, not as a required runtime.\n"
 }
}