{
 "body": "\n## What it measures\n\nUSACO gives a model a competitive-programming problem statement drawn from real USA Computing\nOlympiad contests and asks it to write a Python program that reads input from stdin and writes the\ncorrect output to stdout. The 307 problems span the four official USACO difficulty tiers, bronze\nthrough platinum, so the benchmark exercises a range of algorithmic problem solving: careful\nsimulation and edge-case handling at the easy end, and advanced data structures, dynamic\nprogramming or graph algorithms at the hard end, in addition to producing runnable, correctly\nformatted code.\n\n## How it is scored\n\nA generated solution is extracted from a Python code block and run against 10 to 17 hidden\nstdin/expected-stdout test cases per problem inside a sandboxed environment with CPU time and\nmemory limits matching contest constraints. The problem counts as solved (pass@1) only if the\nprogram's output matches the expected output exactly on every hidden test case; there is no partial\ncredit for passing some but not all tests. The paper's headline numbers use zero-shot chain-of-thought\nprompting and a single sample per problem; it separately studies inference-time methods such as\nself-reflection and retrieval over episodic knowledge, and a human-in-the-loop hint condition, which\nare not directly comparable to the single-sample pass@1 numbers.\n\n## Dataset and licence\n\nThe benchmark's main set holds 307 problems (123 bronze, 100 silver, 63 gold, 21 platinum), each\npaired with its official problem statement, 10 to 17 hidden test cases, a reference solution, and\n(for many problems) the official contest analysis. The authors also released a larger 484-problem\n\"usaco_v2\" set covering all USACO problems with available test cases through September 2023. No\nexplicit licence for the dataset was found in the paper or the GitHub repository read for this\npage; data is distributed via a Google Drive link referenced from the repository rather than\nthrough a licensed dataset host.\n\n## Who publishes it\n\nUSACO was introduced by Quan Shi, Michael Tang, Karthik Narasimhan and Shunyu Yao of the Princeton\nNLP Group in the 2024 paper \"Can Language Models Solve Olympiad Programming?\" The project\nmaintains its own site and GitHub repository, and results are also tracked on Princeton's Holistic\nAgent Leaderboard (HAL).\n\n## Lineage\n\nUSACO is not built as a subset or successor of an earlier benchmark tracked in this repository; it\nadapts an existing, decades-old human competition (the USA Computing Olympiad) into a language\nmodel evaluation set. It has no predecessor or successor page in this repository. inspect_evals\nimplements it as a single task with a configurable dataset version (the 307-problem paper set or\nthe 484-problem extended set) rather than as separate subset pages.\n\n## Saturation and contamination\n\nAt release, GPT-4 solved only 8.7% of problems zero-shot with chain-of-thought prompting, rising to\n20.2% with the paper's best inference-time method, both well short of the reported 35.83% average\nhuman contestant pass rate, so the benchmark was clearly unsaturated in 2024. No maintained,\nindependently confirmed leaderboard tracking current frontier-model scores across all four\ndifficulty tiers was found for this page beyond a listing on the Holistic Agent Leaderboard, so\npresent-day saturation is not established. Contamination risk is medium: the problems are drawn\nfrom publicly archived real contests that predate the paper, and the curated dataset itself\n(statements, hidden tests, reference solutions and official analyses) is openly distributed, so\nmodels trained on contest-archive scrapes or on the released benchmark files could have seen exact\nproblems and solutions.\n\n## How to run it\n\nRun via inspect_evals with the `usaco` task, which loads either the 307-problem paper subset or\nthe 484-problem extended set, executes generated code in a sandboxed Docker environment with CPU\nand memory limits, and reports pass@1 with standard error. The original paper's evaluation harness\nand data are in the `princeton-nlp/USACO` GitHub repository. Numbers are hard to compare across\nreports that differ in prompting strategy (plain zero-shot vs. chain-of-thought vs. retrieval or\nself-reflection methods), number of samples per problem, or which of the two dataset versions was\nused.\n\n## Reading the numbers\n\nA high pass@1 on USACO shows a model can turn an olympiad-style problem statement into working,\nefficient Python code that handles edge cases correctly under real contest constraints, a\nmeaningfully harder bar than typical function-completion coding benchmarks. Because scores are\nreported per difficulty tier, an aggregate number can hide a model that solves nearly all bronze\nproblems but almost none at platinum; check the tier breakdown before treating one score as\nrepresentative. The gap to the 35.83% human average, and the large jump from zero-shot to\ninference-time methods in the original paper, both indicate most gains here as of the paper's\nrelease came from search and self-correction strategies rather than raw model capability alone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "usaco",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "USACOBench"
  ],
  "category": "coding",
  "contamination": {
   "note": "Problems and reference solutions come from real, publicly archived USACO contests dating back years before the paper's release, and the paper's data (problem text, official analyses, reference code) is distributed openly, so exact problem statements and even solutions could appear in training data for models trained on contest-archive scrapes or on the released dataset itself. The 484-problem usaco_v2 extension only goes up to problems with test cases available by September 2023, so newer real USACO contests held after that date are not covered and would be less contaminated if used instead.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 307,
   "size_note": "307 curated problems (the \"usaco_subset307\" set used in the paper's main results): 123 bronze, 100 silver, 63 gold, and 21 platinum, each with 10-17 hidden test cases. The dataset authors also released a larger \"usaco_v2\" set of 484 problems covering all USACO problems with test cases available up to September 2023; inspect_evals supports loading either version but defaults to the 307-problem set used in the paper.\n",
   "splits": "No train/test split; all 307 (or 484) problems are used as a single evaluation set, distributed via a Google Drive link from the project's GitHub repository",
   "url": "https://princeton-nlp.github.io/USACOBench/"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-008 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-008"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "usaco",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "usaco",
  "last_updated": "",
  "leaderboard_url": "https://hal.cs.princeton.edu/usaco",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "USACO gives a model a competitive-programming problem statement, drawn from real USA Computing Olympiad contests, and asks it to write a Python program that reads from standard input and writes to standard output. Problems span four official USACO difficulty tiers (bronze, silver, gold, platinum), so solving them requires algorithmic problem solving, not just syntactic code generation: correct handling of edge cases, algorithmic techniques appropriate to the tier (from basic simulation at bronze to advanced data structures and graph algorithms at platinum), and code that runs within contest time and memory constraints.\n",
  "metric": {
   "baseline_note": "Human baseline of 35.83% average pass@1 is taken from past USACO contest performance, as reported in the paper and echoed in the inspect_evals task metadata. Random baseline is effectively 0 since problems require generated, executable code rather than a choice among options.\n",
   "direction": "higher_is_better",
   "human_baseline": 35.83,
   "max_score": 100.0,
   "name": "pass@1 (fraction of problems solved by a single generated solution passing all hidden tests)",
   "random_baseline": 0.0,
   "unit": "%"
  },
  "name": "USACO",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2404.10952",
   "title": "Can Language Models Solve Olympiad Programming?",
   "url": "https://arxiv.org/abs/2404.10952",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Quan Shi",
    "Michael Tang",
    "Karthik Narasimhan",
    "Shunyu Yao"
   ],
   "org": "Princeton NLP Group",
   "url": "https://princeton-nlp.github.io/USACOBench/"
  },
  "released": "2024-04",
  "repo_url": "https://github.com/princeton-nlp/USACO",
  "saturation": {
   "as_of": "2024-04",
   "note": "At release, GPT-4 scored 8.7% pass@1 zero-shot chain-of-thought and 20.2% with the paper's best inference-time method (self-reflection plus retrieval over episodic knowledge), both far below the 35.83% human contestant average, so the benchmark was clearly unsaturated. No maintained leaderboard tracking current frontier-model scores across all four difficulty tiers was independently confirmed for this page beyond the HAL leaderboard listing, so present-day saturation is not established.\n",
   "status": "open",
   "top_score": 20.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals usaco task directory README (dataset description, 307-problem breakdown by tier, scoring method, example pass@1 figures)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/usaco"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals usaco.py implementation: usaco_subset307 vs usaco_v2 (484 problems) dataset options, sandboxed scorer, code-block extraction",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/usaco/usaco.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals usaco eval.yaml metadata: coding group, human baseline pass@1 0.3583, arxiv 2404.10952 reference",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/usaco/eval.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "Can Language Models Solve Olympiad Programming? (Shi, Tang, Narasimhan, Yao, 2024): 307 problems (123/100/63/21 by tier), 10-17 hidden tests per problem, GPT-4 8.7% zero-shot CoT pass@1, 20.2% best method, human average 35.83%",
    "url": "https://ar5iv.labs.arxiv.org/abs/2404.10952"
   },
   {
    "accessed": "2026-09-08",
    "title": "princeton-nlp/USACO repository README: usaco307 vs usaco_v2 (484 problems, test cases up to September 2023) dataset description and data access via Google Drive",
    "url": "https://github.com/princeton-nlp/USACO"
   },
   {
    "accessed": "2026-09-08",
    "title": "Holistic Agent Leaderboard (HAL) USACO leaderboard page",
    "url": "https://hal.cs.princeton.edu/usaco"
   }
  ],
  "status": "active",
  "subcategory": "olympiad-level competitive programming (algorithmic problem solving)",
  "summary": "A 307-problem benchmark built from USA Computing Olympiad contests that tests whether a model can write a Python program that passes hidden stdin/stdout test cases under time and memory limits.",
  "tags": [
   "coding",
   "competitive-programming",
   "algorithms",
   "agentic"
  ],
  "task_format": "A model receives the problem statement and must produce a Python solution; inspect_evals extracts the code from a ```python markdown block. The solution is executed against 10-17 hidden stdin/expected-stdout test cases per problem inside a sandboxed environment with CPU time and memory limits, and is scored correct only if it matches expected output exactly on every test case. The original paper reports pass@1 under zero-shot chain-of-thought prompting as its main number and also studies inference-time methods (retrieval over episodic knowledge, self-reflection, human-in-the-loop hints).\n"
 }
}