1,000 data-science coding problems across seven Python libraries, perturbed from real StackOverflow questions and checked by execution plus surface-form API constraints.
unassessed
| Category | coding |
|---|---|
| Subcategory | data-science code generation grounded in real StackOverflow problems |
| Page status | active |
| Metric | pass@1 (execution-based, plus surface-form constraint checks) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1000 |
| Dataset licence | CC BY-SA 4.0 (GitHub repository) |
| Publisher | University of Hong Kong, with co-authors at Stanford, UC Berkeley, the University of Washington, Meta AI and Carnegie Mellon University |
DS-1000 tests whether a model can write short, realistic Python code that calls data-science libraries correctly, rather than solving abstract algorithmic puzzles. Each of the 1,000 problems asks for a snippet using one of seven libraries -- NumPy, Pandas, Matplotlib, Scikit-learn, SciPy, PyTorch or TensorFlow -- to do something a real practitioner asked how to do on Stack Overflow, such as reshaping a dataframe or configuring a specific plot. Problems are deliberately perturbed from their original Stack Overflow source (reworded, given a different but equally hard requirement, or made harder) specifically so a model cannot simply reproduce a memorised Stack Overflow answer verbatim and still pass.
Python code generation from a natural-language problem plus surrounding code context (either completion-style or insertion-style), one function or snippet per problem.
No model card in ModelSpec reports this benchmark yet.