OpenAI's benchmark of over 1,400 real Upwork freelance tasks on the Expensify codebase, worth $1 million in actual historical payouts, scored in dollars earned rather than percent resolved.
unassessed
| Category | coding |
|---|---|
| Subcategory | freelance software-engineering tasks and managerial decisions, priced in real dollars |
| Page status | active |
| Metric | $ earned (sum of the payout values of resolved tasks), alongside pass rate per task type |
| Direction | higher_is_better |
| Unit | $ |
| Dataset size | 1488 |
| Dataset licence | MIT |
| Publisher | OpenAI |
SWE-Lancer gives a model real freelance software-engineering work pulled from Upwork job postings against the Expensify open-source repository, with each task tagged at the dollar amount actually paid out for it historically. Two task types are covered: independent engineering (IC SWE) tasks, which range from small bug fixes worth $50 to large feature builds worth up to $32,000 and are graded by running an end-to-end test suite against the model's patch; and SWE Manager tasks, which give the model several competing technical implementation proposals for an issue and ask it to pick the one the real hiring manager chose. Because every task carries its real-world price, a model's performance converts directly into a dollar figure rather than an abstract percentage.
IC SWE: given an issue description and full repository access inside a Docker container, the model edits code and submits a patch, graded by an end-to-end Playwright test suite it cannot see during the attempt; payout is all-or-nothing per task, with no partial credit. SWE Manager: given an issue and several candidate implementation proposals originally written by competing freelancers, the model must select the proposal the real, original engineering manager actually chose.
No model card in ModelSpec reports this benchmark yet.