{
 "body": "\n## What it measures\n\nSWE-bench Lite is the official 300-task reporting subset of original SWE-bench.\nThe agent still receives a GitHub issue and a Python repository snapshot, then\nmust write a patch that the project's tests accept. The subset keeps 11 of the\n12 original repositories but throws away tasks that are long, multi-file, or\npoorly specified.\n\nThe filter is intentional. Lite exists so groups can iterate without running\nall 2,294 full-set instances. That also makes Lite easier than the unfiltered\nset: gold patches edit one file and at most three hunks.\n\n## How it is scored\n\nScoring is unchanged from SWE-bench. A patch must flip FAIL_TO_PASS tests to\npassing and leave PASS_TO_PASS tests passing. OpenAI's Verified announcement\ncited the SWE-bench leaderboard at 43% on Lite as of 2024-08-05, not an OpenAI\nin-house Lite run. The swebench.com Lite board later lists agent results up\nto 60.33% (ExpeRepair-v1.0 + Claude 4 Sonnet, 2025-06-25, unchecked) and 56.67%\nfor a checked SWE-agent + Claude 4 Sonnet run (2025-05-26). Scaffold and whether\nmaintainers checked logs still matter as much as the model name.\n\n## Dataset and licence\n\nThe SWE-bench GitHub repository is MIT-licensed; the Hugging Face Lite cards do\nnot declare a separate licence tag. Hugging Face hosts matching test=300 and\ndev=23 splits under princeton-nlp/SWE-bench_Lite (created 2024-03-19) and\nSWE-bench/SWE-bench_Lite (last modified 2026-08-16). Gold patches and test names\nare public. R2E-Gym/SWE-Bench-Lite is a third 300-row dump with extra Docker\nfields; it is a repack, not a new sample.\n\n## Who publishes it\n\nCarlos E. Jimenez, John Yang, and the original SWE-bench authors at Princeton\nNLP introduced Lite with the SWE-bench project. The SWE-bench Team still hosts\nthe overview at swebench.com/lite.html and the leaderboard at swebench.com.\n\n## Lineage\n\nLite is a subset of [SWE-bench](swe_bench.md), not of\n[SWE-bench Verified](swe_bench_verified.md). Verified later became the default\nfairness-filtered set of 500 tasks and explicitly superseded Lite for most\nvendor reporting. Do not confuse this page with the frozen `lite` split of\n[SWE-bench-Live](swe_bench_live.md), which is a different 300-instance sample\nfrom post-2024 issues.\n\n## Saturation and contamination\n\n60% on a filtered 300-task set is high enough to watch, especially because\nVerified is now the preferred board. Contamination risk is high: these are the\nsame public pull requests as the original benchmark, just the easier slice.\n\n## How to run it\n\nRun the official SWE-bench harness on the Lite dataset id. If you use\ninspect_evals, pass `dataset=\"princeton-nlp/SWE-bench_Lite\"` into\n`inspect_evals/swe_bench`; the default inspect task is Verified. Record whether\nyou used the 23-instance dev split (you should not report it as the test score).\n\n## Reading the numbers\n\nA Lite score is not a Verified score and not a full SWE-bench score. It over-\nrepresents short, single-file Python fixes in well-known projects. When a model\ncard says \"SWE-bench\" without Lite/Verified/full, ask which split. Prefer\nVerified for a current fairness-filtered number, and treat Lite as a cheaper\nhistorical comparison.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_bench_lite",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-bench_Lite",
   "SWE-Bench-Lite"
  ],
  "category": "coding",
  "contamination": {
   "note": "Same structural risk as SWE-bench: public merged pull requests with gold patches in the dataset. Filtering for single-file edits does not hide those patches.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "Python"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 300,
   "size_note": "300 test instances from 11 of the original 12 SWE-bench Python repositories, plus 23 development instances. Hugging Face princeton-nlp/SWE-bench_Lite and SWE-bench/SWE-bench_Lite both expose test=300 and dev=23. Selection removes images and external links, problem statements under 40 words, gold patches that touch more than one file or more than three hunks, file create/delete, and tests that check error strings, then samples 300+23 from the remainder.\n",
   "splits": "test 300, dev 23",
   "url": "https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-074 (Codex coordinated)",
   "reviewed": "2026-09-09",
   "reviewed_by": "Grok Build independent review, batch-074"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Official SWE-bench Docker harness and `swebench` CLI, with dataset princeton-nlp/SWE-bench_Lite or SWE-bench/SWE-bench_Lite. inspect_evals ships inspect_evals/swe_bench (default Verified) and can load Lite if the dataset argument is overridden; there is no dedicated inspect task name for Lite. R2E-Gym/SWE-Bench-Lite is a 300-row Docker-enriched copy, not the official split.\n"
  },
  "id": "swe_bench_lite",
  "last_updated": "2026-08",
  "leaderboard_url": "https://www.swebench.com/",
  "lineage": {
   "family": "swe_bench",
   "predecessor": "swe_bench",
   "successors": [
    "swe_bench_verified"
   ],
   "variants": []
  },
  "measures": "SWE-bench Lite measures the same skill as SWE-bench: produce a patch that resolves a real GitHub issue in a popular Python repository. The 300 test tasks are filtered to shorter, more self-contained edits so evaluation is cheaper than the full 2,294-instance set.\n",
  "metric": {
   "baseline_note": "No human baseline. OpenAI's 2024-08-13 Verified post cited the SWE-bench leaderboard at 43% on Lite as of 2024-08-05. swebench.com later lists higher agent scores (see saturation).",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "% resolved",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SWE-bench Lite",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2310.06770",
   "title": "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?",
   "url": "https://arxiv.org/abs/2310.06770",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Carlos E. Jimenez",
    "John Yang",
    "Alexander Wettig",
    "Shunyu Yao",
    "Kexin Pei",
    "Ofir Press",
    "Karthik R. Narasimhan"
   ],
   "org": "SWE-bench Team (originally Princeton NLP)",
   "url": "https://www.swebench.com/lite.html"
  },
  "released": "2024-03",
  "repo_url": "https://github.com/SWE-bench/SWE-bench",
  "saturation": {
   "as_of": "2025-06",
   "note": "On swebench.com's Lite board (embedded JSON, accessed 2026-09-08) the highest listed resolved rate is 60.33% for ExpeRepair-v1.0 + Claude 4 Sonnet (2025-06-25, unchecked). The highest checked entry among the top rows is SWE-agent + Claude 4 Sonnet at 56.67% (2025-05-26). OpenAI's August 2024 note cited 43% on Lite. The split still sits below 100%, but many vendors now prefer Verified, so Lite is a watch item rather than the default open ranking.\n",
   "status": "watch",
   "top_score": 60.33
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench Lite overview",
    "url": "https://www.swebench.com/lite.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench/SWE-bench_Lite dataset card",
    "url": "https://huggingface.co/datasets/SWE-bench/SWE-bench_Lite"
   },
   {
    "accessed": "2026-09-08",
    "title": "princeton-nlp/SWE-bench_Lite dataset card",
    "url": "https://huggingface.co/datasets/princeton-nlp/SWE-bench_Lite"
   },
   {
    "accessed": "2026-09-08",
    "title": "R2E-Gym/SWE-Bench-Lite (repack, not the official split)",
    "url": "https://huggingface.co/datasets/R2E-Gym/SWE-Bench-Lite"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench leaderboards (Lite JSON)",
    "url": "https://www.swebench.com/"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench paper",
    "url": "https://arxiv.org/abs/2310.06770"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-bench/SWE-bench repository",
    "url": "https://github.com/SWE-bench/SWE-bench"
   },
   {
    "accessed": "2026-09-09",
    "title": "inspect_evals SWE-bench README",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/swe_bench/README.md"
   },
   {
    "accessed": "2026-09-09",
    "title": "OpenAI: Introducing SWE-bench Verified",
    "url": "https://openai.com/index/introducing-swe-bench-verified/"
   },
   {
    "accessed": "2026-09-09",
    "title": "SWE-bench MIT licence",
    "url": "https://raw.githubusercontent.com/SWE-bench/SWE-bench/main/LICENSE"
   }
  ],
  "status": "active",
  "subcategory": "GitHub issue resolution / patch generation (filtered subset)",
  "summary": "A 300-task, single-file-edit subset of original SWE-bench, kept as a cheaper Python issue-resolution reporting split.",
  "tags": [
   "coding",
   "agentic",
   "github-issues",
   "patch-generation",
   "docker",
   "python",
   "subset"
  ],
  "task_format": "Identical to SWE-bench: issue text plus repository access, patch output, Docker grading with FAIL_TO_PASS and PASS_TO_PASS tests. Lite additionally drops multi-file gold patches, file create/delete, short problem statements, and several other hard cases.\n"
 }
}