SciCode decomposes 80 scientist-authored research coding problems into 338 subproblems; a main problem counts solved only when every subproblem and the full integration are correct.
active
Recorded reasons:
| Category | coding |
|---|---|
| Subcategory | scientific research code generation, decomposed into ordered subproblems |
| Page status | active |
| Metric | pass@1 on test-set subproblems (Artificial Analysis); publisher also reports main-problem resolve rate |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 338 |
| Dataset licence | Apache-2.0 |
| Publisher | University of Illinois Urbana-Champaign, Argonne National Laboratory, Carnegie Mellon University, and a multi-institution scientist collaboration |
SciCode asks a model to write Python code that solves real research problems drawn from 16 subfields across five natural-science domains — physics, mathematics, materials science, biology and chemistry — contributed and reviewed by working scientists rather than adapted from textbooks or competitive-programming sites. Each of the 80 main problems is broken into a sequence of smaller subproblems (a median of 3, up to 15), mirroring how a scientist would actually decompose a research task: implement one function, then another, then integrate them into a full solution. Every subproblem and main problem ships with an optional block of scientist-written background knowledge and a docstring specifying the required input/output behaviour, plus gold-standard reference code and test cases. Later subproblems typically depend on earlier ones within the same main problem, so a model's own earlier mistakes can cascade forward.
Code generation, zero-shot: for each subproblem the model is given a docstring (and, in the with-background setting, scientist-written background text) plus code for that main problem's earlier subproblems, and must write the next Python function. A subproblem is judged correct if the generated function passes its held-out test cases. A main problem is judged correct only if every one of its subproblems is correct and the fully integrated solution also passes. Artificial Analysis reports only subproblem pass@1 and always includes the scientist-annotated background.
Each row was checked against its source by a reviewer.
| Model | Score | Evidence date | Source kind | Link |
|---|---|---|---|---|
| GPT-6 Astra (max) | 56.0% | 2026-09-04 published | independent_evaluator | source |
| GLM-5.3 (max) | 59.0% | 2026-09-04 published | independent_evaluator | source |
No model card in ModelSpec reports this benchmark yet.