SWE-Explore isolates repository exploration from patch generation, scoring the ranked code regions an agent returns for a GitHub issue under a fixed line budget.
unassessed
| Category | coding |
|---|---|
| Subcategory | repository exploration / code localization |
| Page status | active |
| Metric | line-level F1 / nDCG@B (with HitFile, HitRegion, context-efficiency and noise-rate as secondary metrics) |
| Direction | higher_is_better |
| Dataset size | 848 |
| Dataset licence | CC BY-NC-ND 4.0 |
Given a repository snapshot and a real GitHub issue, the agent (or retriever) must return a ranked list of code regions it believes are relevant to resolving the issue, subject to a fixed line budget (100, 300 or 500 lines in the released evaluator). The task separates exploration and localization from patch writing, which most issue-resolution benchmarks grade only as a single pass/fail outcome. Ground-truth "core" and "optional context" regions are derived from independent, successful repair trajectories rather than authored by hand.
Text input (issue description plus repository access); output is a ranked list of code regions (file + line range) within the line budget, not a patch.
No model card in ModelSpec reports this benchmark yet.