Open-ended CS problems with continuous partial scoring on algorithmic and research tracks; Inspect pins 238 items while later releases add more.
unassessed
| Category | coding |
|---|---|
| Subcategory | open-ended algorithmic and research CS problems with continuous partial scoring |
| Page status | active |
| Metric | mean per-problem score in [0, 1] (Inspect); paper also reports Score@k / Avg@k / Pass@k |
| Direction | higher_is_better |
| Dataset size | 238 |
| Dataset licence | Hugging Face card Apache-2.0; GitHub LICENSE file MIT (dated 2025). Two licence strings are published. |
| Publisher | FrontierCS team (ICML 2026); paper lists a large multi-institution author list led by Qiuyang Mang and Wenhao Chai |
Frontier-CS asks a model to write an executable program for a computer-science problem whose optimum is unknown, then scores that program with a deterministic checker rather than pass-or-fail unit tests. The December 2025 paper has 156 problems: 107 algorithmic (optimization, constructive, and interactive contest-style tasks, typically C++) and 49 research tasks in operating systems, HPC, AI, databases, programming languages, and security. Inspect Evals pins an expanded Hugging Face snapshot of 238 problems (172 algorithmic + 66 research). English problem statements. GPU research tasks are off by default in Inspect (`include_gpu_problems=False`, 23 of 66 research items).
Inspect default is an agentic ReAct loop with bash and Python, message_limit 100, in Docker. `agentic=False` is single-turn generation, closer to the paper. Track filters: frontier_cs (all), frontier_cs_algorithmic, frontier_cs_research. Algorithmic solutions compile with g++ -O2 -std=gnu++17 and run through testlib checkers. Research solutions run problem-specific Python evaluators.
No model card in ModelSpec reports this benchmark yet.