LiveCodeBench Pro tests models on Olympiad-level Codeforces, ICPC and IOI problems, annotated by competitive-programming medalists, and still finds 0% pass@1 on hard problems for most models.
unassessed
| Category | coding |
|---|---|
| Subcategory | Olympiad-level competitive programming (Codeforces, ICPC, IOI), continuously updated, medalist-annotated |
| Page status | active |
| Metric | Pass@1 by difficulty tier (Easy/Medium/Hard) |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Apache-2.0 per the Hugging Face dataset card's licence tag; no licence file was found in the GavinZhengOI/LiveCodeBench-Pro evaluation-code repository. Individual problem statements originate from Codeforces, ICPC and IOI and may carry those contests' own terms independent of this repackaging. |
| Publisher | New York University, Princeton University, UC San Diego, University of Washington and other institutions (multi-university team with Olympiad-medalist annotators) |
LiveCodeBench Pro asks whether frontier models can solve genuinely hard, Olympiad-level competitive programming problems, in response to claims that LLMs now outperform elite human competitors. Problems come from Codeforces, ICPC and IOI (plus a small number of university contests), captured in real time as contests conclude and before any accepted solutions, editorials or discussion threads appear online, so the pool keeps growing rather than sitting fixed. A team of Olympiad medalists tags every problem by difficulty and by a three-way cognitive-focus taxonomy -- knowledge-heavy (template algorithms and mathematical facts), logic-heavy (systematic derivation) and observation-heavy (problems that need a creative insight or "aha moment") -- and the same medalists conduct line-by-line review of failed model submissions to characterize where models actually go wrong.
The model is given a competitive-programming problem statement and must produce a working solution, typically in C++, graded against hidden test cases by an automated judge; the paper's primary evaluation gives models no external tools or terminal access. Problems are grouped into Easy (Codeforces rating up to 2000), Medium (2000-3000) and Hard (above 3000) difficulty tiers.
No model card in ModelSpec reports this benchmark yet.