{
 "body": "\n## What it measures\n\nCORE-Bench gives an agent a real, previously published research code repository (\"capsule,\" in the\nsource platform's own terminology) and a set of questions about the numeric or visual results that\nrunning that code correctly should produce -- for example, a model's test accuracy after a given\nepoch, or the value on a specific axis of a generated figure. The agent must read instructions,\ninstall the right dependencies, run the code, and report an answer for every question tied to that\npaper (at the easiest difficulty level, it is instead given the output already and only needs to\nextract from it). Papers span computer science, social science and medicine, written in Python or R,\nand some questions require reading a generated chart or table image rather than only text. Unlike\nsingle-turn code-generation or question-answering benchmarks, this measures a genuinely agentic\nskill: multi-step tool use, debugging and retrieval inside a real, occasionally uncooperative\nsoftware environment.\n\n## How it is scored\n\nAn agent writes its final answers to a report.json file; a task counts as solved only if every\nquestion associated with that paper is answered correctly, with no partial credit. Numeric answers\nare checked against a tolerance, other answers by a normalized match. The three difficulty levels\ngive progressively less help: CORE-Bench-Easy hands the agent the output of an already-successful\nrun and asks only for information extraction; CORE-Bench-Medium provides a Dockerfile and README\ninstructions, so the agent need only run the given command and then extract results; CORE-Bench-Hard\nprovides only a README and no Dockerfile, requiring the agent to work out and install the correct\ndependencies itself before running anything. The authors designed each task to include at least one\nquestion that cannot be answered by guessing, so there is no meaningful random baseline, and no\nhuman-reproducibility baseline percentage was found in the sources reviewed for this page.\n\n## Dataset and licence\n\nThe benchmark comprises 270 tasks from 90 papers (three difficulty levels per paper), with 181\ndistinct task questions in total, since the same questions repeat across a paper's three difficulty\nlevels. The 90 papers were filtered from 5,090 candidate capsules on CodeOcean.com against ten\nselection criteria, chief among them that the authors could verify the capsule was locally\nreproducible. The 90 papers split 45/45 into training and test sets (135 tasks each); the standard\n`core_bench` evaluation runs only against the 45-paper test half. That test half is distributed\nPGP-encrypted, with the decryption password published in the reference repository's own README -- a\nlight deterrent against casual scraping rather than genuine secrecy. The dataset and code are\nreleased under the MIT licence.\n\n## Who publishes it\n\nCORE-Bench was introduced by Zachary S. Siegel, Sayash Kapoor, Nitya Nadgir, Benedikt Stroebl and\nArvind Narayanan of Princeton University, posted to arXiv in September 2024. The same group\nmaintains the dataset on Hugging Face and, more recently, the Holistic Agent Leaderboard (HAL), a\nbroader multi-benchmark agent leaderboard that has become the recommended way to run and report\nCORE-Bench results; the original standalone harness's own README now states it \"is no longer\nactively maintained.\"\n\n## Lineage\n\nCORE-Bench has no named predecessor or formal successor benchmark. Its most direct continuation is\norganizational rather than a new id: the same Princeton group folded CORE-Bench into the Holistic\nAgent Leaderboard, a shared harness covering multiple agentic benchmarks, superseding the original\nstandalone evaluation harness though not the underlying task or dataset. This repository does not\nhave separate pages for the three difficulty levels, run as parameters of one task rather than as\nseparate benchmarks.\n\n## Saturation and contamination\n\nThe paper's own headline result is that the best baseline agent (a task-specific CORE-Agent running\nGPT-4o) reached only 21% task accuracy on CORE-Bench-Hard, \"showing the vast scope for improvement.\"\nThat figure is now over two years old and predates most current frontier agent scaffolds; the\nHolistic Agent Leaderboard, the successor leaderboard the reference repository now points to,\nreturned an error page rather than rendering results when checked for this page (2026-09-08), so no\ncurrent top score could be confirmed. Given how far even a capable contemporary agent sat below full\naccuracy, this benchmark is graded \"open.\" Contamination risk is graded low: the paper explicitly\nargues that drawing from public repositories lets the benchmark be refreshed over time to manage\ncontamination, and the test split's specific question/answer pairs are distributed encrypted rather\nthan in plain text, even though the underlying research code itself was already public before\nselection.\n\n## How to run it\n\ninspect_evals implements this as a single `core_bench` task with a `difficulty` parameter (easy,\nmedium or hard) plus filters for field, language and GPU requirement; running it downloads and\nPGP-decrypts the test split and requires Docker-in-Docker sandboxing for medium/hard tasks that\nexecute code, which in turn can require a GPU. The original reference harness (siegelz/core-bench)\ncapped agents at $4 of API spend per task and evaluated a general-purpose AutoGPT agent alongside a\ntask-specific CORE-Agent, both with GPT-4o and GPT-4o-mini; that harness is no longer actively\nmaintained in favor of the Holistic Agent Leaderboard harness from the same authors. No lm-eval-harness, HELM, OpenCompass or BIG-bench implementation was found.\n\n## Reading the numbers\n\nA high CORE-Bench score shows an agent can navigate an unfamiliar, real code repository well enough\nto install its dependencies, execute it correctly, and pull out the right numbers or figure\nreadings -- a close proxy for whether it could actually help verify or replicate published research,\nrather than merely writing plausible-looking code. Because a task requires every associated question\nto be answered correctly, scores are stricter than a per-question accuracy would suggest, and\ncomparing scores across difficulty levels shows how much of an agent's success depends on being\nhanded a working environment versus building one itself. Given the benchmark's low original ceiling\nand the unconfirmed state of current leaderboard data, treat any single number as provisional until\nchecked against the harness, model and difficulty level that produced it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "core_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Computational Reproducibility Benchmark"
  ],
  "category": "agentic",
  "contamination": {
   "note": "The paper states directly that \"CORE-Bench's foundation in public repositories enables periodic updates of the benchmark tasks, which could mitigate concerns about contamination and saturation,\" and the test split's task questions and reference answers are distributed PGP-encrypted rather than in plain text, unlike the underlying CodeOcean code itself (which was already public before selection). That encryption is a soft deterrent, since its password is published in the reference repository, but it does prevent the specific question/answer pairs from appearing in a plain web crawl.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "code",
    "image"
   ],
   "public_test_set": true,
   "size": 270,
   "size_note": "270 tasks drawn from 90 papers (three difficulty levels per paper: easy, medium, hard), each with at least one question -- 181 distinct task questions in total, since the same questions repeat across a paper's three difficulty levels. The 90 papers were filtered from 5,090 candidate CodeOcean.com capsules against ten selection criteria (local reproducibility chief among them), confirmed directly from the paper's own description of this process. The dataset splits the 90 papers 45/45 into training and test papers (135 tasks each); the `core_bench` task most harnesses run evaluates only the 45-paper test half. The test split ships PGP-encrypted; the decryption password is published in the reference repository's own README, so this is better read as a light deterrent against casual scraping than a true secrecy mechanism.\n",
   "splits": "45 papers / 135 tasks for training, 45 papers / 135 tasks for test (encrypted); most reported scores use only the test half.\n",
   "url": "https://huggingface.co/datasets/siegelz/core-bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "core_bench (single task, `difficulty` parameter: easy | medium | hard, default easy; confirmed directly in the inspect_evals source)",
   "lm_eval": "",
   "opencompass": "",
   "other": "The reference implementation (siegelz/core-bench on GitHub) states in its own README that it \"is no longer actively maintained\" and directs users to the Holistic Agent Leaderboard harness (princeton-pli/hal-harness) instead, from the same research group. The original baselines (AutoGPT and a task-specific CORE-Agent, both run with GPT-4o and GPT-4o-mini) were capped at $4 of API spend per task. Medium and hard tasks that need a GPU require Docker-in-Docker sandboxing; inspect_evals' implementation notes it can only run every task without a GPU at the easy difficulty level, since that level requires no code execution."
  },
  "id": "core_bench",
  "last_updated": "2024-10",
  "leaderboard_url": "https://agent-evals-leaderboard.hf.space",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "CORE-Bench gives an agent a real, previously published research repository (\"capsule,\" in the source platform's terminology) and a set of questions about the numeric or visual results that running that code correctly should produce -- for example, a model's test accuracy after a given epoch, or the value of a specific axis label on a generated figure. The agent must read instructions, install the right dependencies, run the code (or, at the easiest level, skip straight to reading output that is already provided), and report answers for every question tied to that paper. Papers span computer science, social science and medicine, written in Python or R, and some questions require reading a generated chart or table image rather than only text. It measures a genuinely agentic skill -- multi-step tool use, debugging, and retrieval inside a real software environment -- rather than single-turn code generation or question answering.\n",
  "metric": {
   "baseline_note": "Task accuracy is the share of tasks where every associated question was answered correctly; numeric answers are checked against a tolerance and non-numeric answers by normalized match. The authors designed each task to include at least one question that cannot be solved by guessing, so no single random-guess percentage applies across the benchmark, and no human-reproducibility baseline percentage was found in the sources reviewed for this page. The paper also reports average API cost per agent run alongside accuracy, since the original baselines were capped at $4 per task.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "task accuracy (all task questions correct)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "CORE-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2409.11363",
   "title": "CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark",
   "url": "https://arxiv.org/abs/2409.11363",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Zachary S. Siegel",
    "Sayash Kapoor",
    "Nitya Nadgir",
    "Benedikt Stroebl",
    "Arvind Narayanan"
   ],
   "org": "Princeton University",
   "url": "https://github.com/siegelz/core-bench"
  },
  "released": "2024-09",
  "repo_url": "https://github.com/siegelz/core-bench",
  "saturation": {
   "as_of": "2024-09",
   "note": "The paper's own headline result -- the best baseline agent (CORE-Agent with GPT-4o) reached 21% task accuracy on CORE-Bench-Hard -- is over two years old at the time of this research and predates most current frontier agent scaffolds. The successor leaderboard the reference repository now points to, the Holistic Agent Leaderboard (agent-evals-leaderboard.hf.space), returned an error page rather than rendering results when checked on 2026-09-08 (\"Your space is in error\"), so no current top score could be confirmed. Given how far even a capable 2024 agent sat below full accuracy, and no confirmed evidence the gap has since closed, this is graded \"open\" rather than \"watch\" or \"saturated.\"\n",
   "status": "open",
   "top_score": 21.0
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark",
    "url": "https://arxiv.org/abs/2409.11363"
   },
   {
    "accessed": "2026-09-08",
    "title": "siegelz/core-bench repository (README, LICENSE)",
    "url": "https://github.com/siegelz/core-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "siegelz/core-bench dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/siegelz/core-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "siegelz/core-bench, Hugging Face Hub API (licence tag, last-modified date)",
    "url": "https://huggingface.co/api/datasets/siegelz/core-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals: core_bench task implementation and README",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/core_bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "Holistic Agent Leaderboard (current recommended CORE-Bench leaderboard; returned an error page on access)",
    "url": "https://agent-evals-leaderboard.hf.space"
   }
  ],
  "status": "active",
  "subcategory": "computational reproducibility: scientific code execution and result verification",
  "summary": "Tests whether an agent can reproduce a published paper's results by installing dependencies, running its code, and extracting the right numbers, across three levels of given scaffolding.",
  "tags": [
   "agentic",
   "code-execution",
   "reproducibility",
   "tool-use",
   "scientific-research",
   "multi-step"
  ],
  "task_format": "Given a paper's code repository and a fixed set of questions about its outputs, the agent works inside an isolated container (with bash and, for image-based questions, a vision-language-model tool) and writes its final answers to a report.json file. A task counts as solved only if every question tied to that paper is answered correctly; there is no partial credit for a task.\n"
 }
}