OpenCompass's interactive benchmark for LLM code-interpreter agents -- 234 multi-step data-science tasks (1,900+ questions) across ten Python libraries, scored end-to-end and in an error-corrected oracle mode.
unassessed
| Category | agentic |
|---|---|
| Subcategory | code-interpreter tool use across interactive, multi-step data-science notebooks |
| Page status | active |
| Metric | process-oriented (tool-call rate, executable rate) and output-oriented (numeric accuracy, text score, visualization score) metrics |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 234 |
| Dataset licence | Apache License 2.0, per the open-compass/CIBench GitHub repository's licence |
| Publisher | Shanghai Artificial Intelligence Laboratory, with ShanghaiTech University |
CIBench tests whether an LLM agent can use a Python code interpreter to carry out a realistic, multi-step data-science workflow, rather than just produce an isolated code snippet for one self-contained problem. Each task is a simulated, interactive IPython/Jupyter session of 10 to 15 progressive steps of increasing complexity, built around one of ten widely used Python libraries (lightgbm, matplotlib, nltk, opencv, pandas, pytorch, scipy, seaborn, sklearn, tensorflow), with a Chinese-language translation of the same task set, plus a separate, smaller six-library open-ended "generation" split (matplotlib, opencv, pandas, pytorch, scipy, seaborn). Tasks were built with an LLM-human cooperative method: an advanced LLM (GPT-4) proposed instructions and code in notebook form, and human experts then wrote reusable template tasks from the common patterns observed, so each template can be instantiated over multiple interchangeable underlying datasets. It is a multi-turn, tool-use task combining natural-language instructions, generated code, and the code interpreter's own output (including numbers, text and rendered plots) at each step.
A sequence of 10 to 15 interconnected natural-language instructions per task, each requiring the model to write and execute Python code in a live IPython kernel and use the returned output to inform its next step; evaluated in two modes -- end-to-end, where the model must solve every step unaided, and oracle, where the model is handed the correct code whenever it fails a step, which isolates a step's individual difficulty from upstream error compounding.
No model card in ModelSpec reports this benchmark yet.