A 273-task continual-learning reformulation of SWE-bench Verified: eight chronological repository sequences with forgetting and transfer metrics.
unassessed
| Category | coding |
|---|---|
| Subcategory | continual learning over SWE-bench Verified sequences |
| Page status | proposed |
| Metric | average accuracy (plus forgetting, transfer, and CL-Score) |
| Direction | higher_is_better |
| Dataset size | 273 |
| Dataset licence | MIT |
| Publisher | Independent course project (COMS 4995) |
SWE-Bench-CL asks whether a coding agent improves, transfers, and avoids forgetting as it walks a stream of GitHub issues from one repository. The underlying issues are SWE-bench Verified tasks, reordered into curricula instead of scored as i.i.d. bugs.
Eight repository sequences (273 tasks total) are ordered first by issue creation time, then by human-estimated fix time. After each task, evaluation can re-test earlier tasks. The authors also describe a LangGraph agent with FAISS memory as a reference scaffold, not as a required runtime.
No model card in ModelSpec reports this benchmark yet.