A 307-problem benchmark built from USA Computing Olympiad contests that tests whether a model can write a Python program that passes hidden stdin/stdout test cases under time and memory limits.
unassessed
| Category | coding |
|---|---|
| Subcategory | olympiad-level competitive programming (algorithmic problem solving) |
| Page status | active |
| Metric | pass@1 (fraction of problems solved by a single generated solution passing all hidden tests) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 307 |
| Publisher | Princeton NLP Group |
USACO gives a model a competitive-programming problem statement, drawn from real USA Computing Olympiad contests, and asks it to write a Python program that reads from standard input and writes to standard output. Problems span four official USACO difficulty tiers (bronze, silver, gold, platinum), so solving them requires algorithmic problem solving, not just syntactic code generation: correct handling of edge cases, algorithmic techniques appropriate to the tier (from basic simulation at bronze to advanced data structures and graph algorithms at platinum), and code that runs within contest time and memory constraints.
A model receives the problem statement and must produce a Python solution; inspect_evals extracts the code from a ```python markdown block. The solution is executed against 10-17 hidden stdin/expected-stdout test cases per problem inside a sandboxed environment with CPU time and memory limits, and is scored correct only if it matches expected output exactly on every test case. The original paper reports pass@1 under zero-shot chain-of-thought prompting as its main number and also studies inference-time methods (retrieval over episodic knowledge, self-reflection, human-in-the-loop hints).
No model card in ModelSpec reports this benchmark yet.