10,000 Python coding problems scraped from competitive-programming and interview sites across three difficulty tiers, graded by executing generated code against held-out test cases.
unassessed
| Category | coding |
|---|---|
| Subcategory | competitive programming, test-case graded |
| Page status | superseded |
| Metric | strict accuracy / pass@k (fraction of problems whose generated code passes every held-out test case) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 10000 |
| Dataset licence | CC BY-SA 3.0 (dataset, per the paper, following source site Kattis's problem licence); the evaluation code is released separately under MIT, and the community codeparrot/apps mirror on Hugging Face is tagged MIT rather than CC BY-SA -- the two licence statements disagree and no source consulted resolves the conflict. |
APPS gives a model a natural-language description of a programming problem -- for interview- and competition-level problems, this includes a formal input/output specification, similar to what a competitive-programming judge shows a human contestant -- and the model must write a complete Python program that solves it. Problems are pulled from real competitive-programming and coding-interview sites rather than authored for the benchmark, and range from problems solvable with a few lines of code to substantial algorithmic challenges. The task mirrors how software-engineering candidates are screened: read a spec, write correct code, with no partial credit for code that merely looks plausible.
Free-form Python code generation from a natural-language problem statement (plus a formal input/output spec for interview- and competition-level problems); not code completion.
No model card in ModelSpec reports this benchmark yet.