{
 "body": "\n## What it measures\n\nSWE-Lancer asks whether a model can do real, paid freelance software-engineering work rather than a\nsynthetic proxy for it. Every one of its 1,488 tasks is a real Upwork job posted against Expensify, an\nopen-source expense-management product, tagged with the dollar amount actually paid for it historically.\nIC SWE tasks are individual-contributor work -- from $50 bug fixes to $32,000 feature builds -- where the\nmodel edits the repository directly. SWE Manager tasks instead give the model several competing\nimplementation proposals for an issue, written by real freelancers, and ask it to pick the one the hiring\nmanager actually chose, probing technical judgment rather than the ability to write code.\n\n## How it is scored\n\nIC SWE tasks are graded by an end-to-end test suite -- largely Playwright browser tests -- that OpenAI\nsays were hand-written and triple-verified by professional engineers rather than reused from the original\npull requests; the model cannot see these tests, and a task pays out in full only if all of them pass,\nwith no partial credit. SWE Manager tasks are scored by exact match against the manager's historical\nchoice among the candidate proposals. OpenAI reports both a percent-resolved pass rate per task type and\nthe total dollar value earned; because dollar value is not spread evenly across tasks, the two can\ndiverge -- a model that solves many cheap tasks but few expensive ones scores well on pass rate yet poorly\nin dollars, and vice versa.\n\n## Dataset and licence\n\nThe full set is 1,488 tasks worth $1,000,000: 764 IC SWE tasks ($414,775) and 724 SWE Manager tasks\n($585,225), drawn from Expensify's real Upwork job history in 2023-2024. OpenAI publicly released a\n502-task subset, SWE-Lancer Diamond ($500,800: 237 IC SWE tasks worth $236,300, 265 SWE Manager tasks\nworth $264,500), with a unified Docker image; the remaining roughly 986 tasks (~$499,200) are held out\nprivately as a contamination guard. The code and dataset in github.com/openai/frontier-evals are\nMIT-licensed. A July 2025 update removed the internet-access requirement during grading and, in adapting\nIC SWE Diamond to run offline, dropped 39 of its 237 tasks that could not work reliably without network\naccess, leaving 198 usable today (see How to run it).\n\n## Who publishes it\n\nSWE-Lancer comes from Samuel Miserendino, Michele Wang, Tejal Patwardhan and Johannes Heidecke at OpenAI,\nannounced 2025-02-18 alongside the arXiv paper. OpenAI updated the dataset and results on 2025-07-17 and\nfolded the evaluation code into its broader `frontier-evals` repository (the original\n`openai/SWELancer-Benchmark` repository is now archived and read-only).\n\n## Lineage\n\nSWE-Lancer is not part of the `swe_bench` family catalogued elsewhere in this repository, but answers a\nsimilar question by a different construction: where `swe_bench` and `swe_bench_verified` mine\nalready-merged GitHub pull requests and grade against the tests recovered from them, SWE-Lancer sources\ntasks from paid Upwork postings, prices each at what a human was actually paid, and adds a second,\nnon-coding task type (SWE Manager) SWE-bench has no equivalent for. Both are container-graded,\nreal-repository benchmarks sharing the same structural contamination risk, but SWE-Lancer's dollar pricing\nand manager-decision tasks make it a distinct instrument rather than a SWE-bench variant. No predecessor,\nsuccessor or catalogued variant of SWE-Lancer itself was identified.\n\n## Saturation and contamination\n\nAt release, OpenAI's own framing was that frontier models \"are still unable to solve the majority of\ntasks\": the best of the three models it reported, Claude 3.5 Sonnet, earned about $208k of the $500,800\navailable on Diamond, with o1 (high) and GPT-4o further behind -- a wide, unsaturated gap; no current\ncross-model leaderboard was located to say where frontier models sit as of 2026. Contamination risk is\nmedium: IC SWE tasks are real, publicly resolved issues, but two-thirds of the full benchmark is held out\nprivately and the grading tests are hidden from the model, both limiting how much memorising a public fix\nwould help.\n\n## How to run it\n\ninspect_evals implements the benchmark as `swe_lancer`, selecting `ic_swe`, `swe_manager` or `all` tasks\nand running each inside a pre-built, per-issue Docker image from Docker Hub; OpenAI's own runner in\n`github.com/openai/frontier-evals/project/swelancer` supports both per-task images and a single \"monolith\"\nimage (required for SWE Manager tasks). The July 2025 update disables internet access during grading --\nOpenAI considers results with internet enabled invalid -- and only 198 of the original 237 IC SWE Diamond\ntasks were adapted to run offline, so a reported score today may cover a smaller item set than the\npaper's original figure; check the task count and repository revision before comparing scores.\n\n## Reading the numbers\n\nA high dollar total on SWE-Lancer is evidence a model can do freelance-grade engineering work a real\nclient would pay for and a real manager would judge sound -- a step beyond an isolated coding puzzle.\nBecause the metric is priced rather than percentage-based, check whether a figure is a dollar total or a\npass rate, and which slice (full set, Diamond, IC SWE only, SWE Manager only) it covers: a high dollar\ntotal can come from solving a few expensive tasks rather than being broadly reliable, which a pass rate\nwould show but a dollar figure alone can obscure. Like SWE-bench, it says little about codebases or\nlanguages outside Expensify's JavaScript/PHP application.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_lancer",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SWE-Lancer Diamond"
  ],
  "category": "coding",
  "contamination": {
   "note": "Every IC SWE task is a real, historically resolved Upwork issue against the public Expensify open-source repository, so its original fix is plausibly discoverable in training data, similar to SWE-bench's contamination profile. Risk is not marked high because roughly two-thirds of the benchmark by task count is held out privately rather than published, specifically to blunt this risk, and the end-to-end grading tests themselves are hidden from the model during the attempt.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "MIT",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": false,
   "size": 1488,
   "size_note": "1,488 freelance tasks worth $1,000,000 total, per OpenAI's published breakdown: 764 IC SWE tasks worth $414,775 and 724 SWE Manager tasks worth $585,225. Only part of this is public: SWE-Lancer Diamond, 502 tasks worth $500,800 (237 IC SWE tasks worth $236,300 and 265 SWE Manager tasks worth $264,500), released with a unified Docker evaluation image; the remaining roughly 986 tasks (~$499,200) are held out privately to limit contamination. A July 2025 dataset update removed the requirement for internet access during execution and, in the process, dropped 39 of the original 237 IC SWE Diamond tasks that could not be adjusted to run reliably offline, leaving 198 of that subset in the current offline-runnable release (see How to run it).\n",
   "splits": "Diamond (public, 502 tasks) vs. the full 1,488-task set (remainder held out privately); ic_swe vs. swe_manager task types within each",
   "url": "https://github.com/openai/frontier-evals/tree/main/project/swelancer"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "swe_lancer",
   "lm_eval": "",
   "opencompass": "",
   "other": "OpenAI's own evaluation code and Docker images live in github.com/openai/frontier-evals (project/swelancer; originally published as the now-archived, read-only openai/SWELancer-Benchmark repository). inspect_evals implements it as `swe_lancer`, with a `task_variant` parameter selecting `ic_swe`, `swe_manager` or `all`, running each task inside a per-issue Docker image pulled from Docker Hub.\n"
  },
  "id": "swe_lancer",
  "last_updated": "2025-07",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-Lancer gives a model real freelance software-engineering work pulled from Upwork job postings against the Expensify open-source repository, with each task tagged at the dollar amount actually paid out for it historically. Two task types are covered: independent engineering (IC SWE) tasks, which range from small bug fixes worth $50 to large feature builds worth up to $32,000 and are graded by running an end-to-end test suite against the model's patch; and SWE Manager tasks, which give the model several competing technical implementation proposals for an issue and ask it to pick the one the real hiring manager chose. Because every task carries its real-world price, a model's performance converts directly into a dollar figure rather than an abstract percentage.\n",
  "metric": {
   "baseline_note": "The headline metric is a dollar figure, not a percentage: a model's score is the sum of the real historical Upwork payout values of the tasks it resolves, so it is weighted toward the rarer, expensive tasks rather than treating every task equally the way a plain pass rate would -- OpenAI reports both the dollar total and a per-task-type pass rate (percent resolved) side by side for exactly this reason. The achievable maximum depends on which slice is run -- $1,000,000 across the full 1,488-task set, $500,800 on the public Diamond split, or a smaller figure for IC SWE or SWE Manager tasks alone -- so max_score is left unset here rather than tied to one slice. No formal human pass-rate baseline is published; each task's dollar value is itself the real amount historically paid to the freelancer who completed it on Upwork, which is a different kind of reference point than a controlled human-solve study and is not treated as one here.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "$ earned (sum of the payout values of resolved tasks), alongside pass rate per task type",
   "random_baseline": 0,
   "unit": "$"
  },
  "name": "SWE-Lancer",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2502.12115",
   "title": "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?",
   "url": "https://arxiv.org/abs/2502.12115",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Samuel Miserendino",
    "Michele Wang",
    "Tejal Patwardhan",
    "Johannes Heidecke"
   ],
   "org": "OpenAI",
   "url": "https://openai.com/index/swe-lancer/"
  },
  "released": "2025-02",
  "repo_url": "https://github.com/openai/frontier-evals",
  "saturation": {
   "as_of": "",
   "note": "At release (February 2025), OpenAI's own headline finding was that \"frontier models are still unable to solve the majority of tasks.\" Of the three models reported on the Diamond set, Claude 3.5 Sonnet earned the most, roughly $208k of the $500,800 available (44.9% pass rate on SWE Manager tasks, 26.2% on IC SWE tasks); o1 (high reasoning effort) earned about $166k and GPT-4o about $139k -- all well short of the ceiling. No current cross-model leaderboard was found during this research to confirm a 2026 top score, so status is read as open on the evidence available rather than watch or saturated.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-Lancer: Can Frontier LLMs Earn $1 Million from Real-World Freelance Software Engineering?",
    "url": "https://arxiv.org/abs/2502.12115"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWE-Lancer paper, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2502.12115"
   },
   {
    "accessed": "2026-09-08",
    "title": "Introducing the SWE-Lancer benchmark | OpenAI",
    "url": "https://openai.com/index/swe-lancer/"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/SWELancer-Benchmark (archived; superseded by openai/frontier-evals)",
    "url": "https://github.com/openai/SWELancer-Benchmark"
   },
   {
    "accessed": "2026-09-08",
    "title": "openai/frontier-evals, project/swelancer -- current repository, README and MIT LICENSE.md",
    "url": "https://github.com/openai/frontier-evals/tree/main/project/swelancer"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals swe_lancer task README",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/swe_lancer"
   }
  ],
  "status": "active",
  "subcategory": "freelance software-engineering tasks and managerial decisions, priced in real dollars",
  "summary": "OpenAI's benchmark of over 1,400 real Upwork freelance tasks on the Expensify codebase, worth $1 million in actual historical payouts, scored in dollars earned rather than percent resolved.",
  "tags": [
   "coding",
   "agentic",
   "freelance",
   "dollar-denominated",
   "upwork",
   "docker",
   "managerial-decision"
  ],
  "task_format": "IC SWE: given an issue description and full repository access inside a Docker container, the model edits code and submits a patch, graded by an end-to-end Playwright test suite it cannot see during the attempt; payout is all-or-nothing per task, with no partial credit. SWE Manager: given an issue and several candidate implementation proposals originally written by competing freelancers, the model must select the proposal the real, original engineering manager actually chose.\n"
 }
}