Tests whether an agent can reproduce a published paper's results by installing dependencies, running its code, and extracting the right numbers, across three levels of given scaffolding.
unassessed
| Category | agentic |
|---|---|
| Subcategory | computational reproducibility: scientific code execution and result verification |
| Page status | active |
| Metric | task accuracy (all task questions correct) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 270 |
| Dataset licence | MIT |
| Publisher | Princeton University |
CORE-Bench gives an agent a real, previously published research repository ("capsule," in the source platform's terminology) and a set of questions about the numeric or visual results that running that code correctly should produce -- for example, a model's test accuracy after a given epoch, or the value of a specific axis label on a generated figure. The agent must read instructions, install the right dependencies, run the code (or, at the easiest level, skip straight to reading output that is already provided), and report answers for every question tied to that paper. Papers span computer science, social science and medicine, written in Python or R, and some questions require reading a generated chart or table image rather than only text. It measures a genuinely agentic skill -- multi-step tool use, debugging, and retrieval inside a real software environment -- rather than single-turn code generation or question answering.
Given a paper's code repository and a fixed set of questions about its outputs, the agent works inside an isolated container (with bash and, for image-based questions, a vision-language-model tool) and writes its final answers to a report.json file. A task counts as solved only if every question tied to that paper is answered correctly; there is no partial credit for a task.
No model card in ModelSpec reports this benchmark yet.