An agent benchmark of verifiable scRNA-seq analysis problems; inspect_evals ships a 30-task public canonical slice, while the paper describes 394 held-out tasks.
unassessed
| Category | agentic |
|---|---|
| Subcategory | agentic single-cell RNA-seq analysis on .h5ad snapshots with deterministic graders |
| Page status | active |
| Metric | accuracy (fraction of tasks passed); stderr also reported |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 394 |
| Dataset licence | Apache-2.0 |
| Publisher | LatchBio |
scBench (LatchBio) asks an agent to analyse a real single-cell RNA-seq snapshot stored as AnnData `.h5ad` and write structured answers to `eval_answer.json`. Tasks cover quality control, normalization, dimensionality reduction, clustering, cell typing, differential expression, and trajectory analysis across sequencing platforms. Graders are deterministic (numeric tolerance, multiple choice, marker-gene precision/recall, label-set Jaccard, distribution comparison). The agent must load the data; a memorised textbook answer fails if it never touches the file. inspect_evals, the census harness for this id, runs the public canonical subset (30 tasks, five platforms), not the full paper set.
Agentic code execution in a sandbox. inspect_evals defaults to mini-SWE-agent via inspect-swe inside Docker, with a react()+bash fallback. Timeout defaults: 600s task, 300s bash. Answers are read only from eval_answer.json; there is no text-parsing fallback.
No model card in ModelSpec reports this benchmark yet.