Agents must replicate 20 ICML 2024 Spotlight and Oral papers from scratch against author-written rubrics totaling 8,316 leaf criteria.
unassessed
| Category | agentic |
|---|---|
| Subcategory | from-scratch replication of ML research papers |
| Page status | active |
| Metric | replication score (weighted fraction of satisfied rubric leaves) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 20 |
| Dataset licence | MIT |
| Publisher | OpenAI |
PaperBench gives an agent a paper PDF or Markdown file, an author addendum, and a sandbox with code execution. The agent must write a repository from scratch, including a reproduce.sh entrypoint, that reimplements the paper's empirical work. It may not use the authors' original code (blacklisted URLs). After the attempt, a fresh machine runs reproduce.sh, then an LLM judge scores the executed submission against a hierarchical rubric co-written with a paper author. A lighter Code-Dev variant skips execution and scores only code-development leaves.
Long-horizon agent task. Output is a codebase plus reproduce.sh. The paper caps reproduce.sh at 12 hours in its experiments and does not cap agent runtime. inspect_evals defaults both agent time and reproduce.sh to 6 hours. Grading is a weighted tree of binary leaf checks (code development, execution, result match) rolled up to one replication score.
No model card in ModelSpec reports this benchmark yet.