{
 "body": "\n## What it measures\n\nDS-1000 tests whether a model can write short, realistic Python code that calls data-science libraries correctly, rather than solving abstract algorithmic puzzles the way HumanEval-style benchmarks do. Each of the 1,000 problems asks for a snippet using one of seven libraries -- NumPy, Pandas, Matplotlib, Scikit-learn, SciPy, PyTorch or TensorFlow -- to do something a real practitioner asked how to do on Stack Overflow, such as reshaping a dataframe in a specific way or configuring a particular kind of plot. Problems are deliberately perturbed from their original Stack Overflow source: reworded, given a different but comparably hard requirement, or rewritten to be harder still, specifically so a model cannot pass just by reproducing a memorised Stack Overflow answer verbatim.\n\n## How it is scored\n\nDS-1000 uses a multi-criteria check rather than plain string matching. Functional correctness runs the generated code against test cases -- 1.6 per problem on average -- and compares actual output to a reference, using statistical tests where results involve floating-point precision or randomness. A separate surface-form check applies to about 19.4% of problems, which additionally require or forbid specific API usage or keywords, such as rejecting an explicit for-loop where a vectorised call is expected even if the loop produces the right output. The paper validated this combined check by having human annotators review whether Codex-002 solutions that passed automated scoring were genuinely correct, finding a false-discovery rate of only 1.8%. The paper's own headline result, \"the current best public system (Codex-002) achieves 43.3% accuracy,\" is now a 2022-era reference point rather than a current one.\n\n## Dataset and licence\n\nThe benchmark holds 1,000 problems, unevenly spread across its seven libraries: Pandas (291), NumPy (220), Matplotlib (155), Scikit-learn (115), SciPy (106), PyTorch (68) and TensorFlow (45). They derive from 451 unique Stack Overflow questions, combined with three perturbation strategies applied on top of an original, closer-to-source version: surface perturbation (152 problems, rewording or reformatting without changing the required solution), semantic perturbation (235 problems, changing the actual requirement while keeping difficulty comparable) and difficult rewrites (162 problems, combining perturbations to raise difficulty). The GitHub repository states a CC BY-SA 4.0 licence; each problem ships with its reference solution, starter code and test functions (`test_execution` and `test_string`), all public with no held-out portion.\n\n## Who publishes it\n\nDS-1000 comes from Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Scott Wen-tau Yih, Daniel Fried, Sida Wang and Tao Yu, whose affiliations at publication spanned the University of Hong Kong, Stanford, UC Berkeley, the University of Washington, Meta AI and Carnegie Mellon University. It appeared on arXiv in November 2022 and at ICML 2023. The reference repository is hosted under `xlang-ai` on GitHub, the dataset is mirrored on Hugging Face as `xlangai/DS-1000`, and a project page at `ds1000-code-gen.github.io` hosts examples and links to an evaluation leaderboard.\n\n## Lineage\n\nDS-1000 has no predecessor or successor tracked in this repository. It sits alongside general-purpose code-generation benchmarks such as HumanEval and MBPP, both in this repository, but targets library-specific data-science code rather than self-contained algorithmic functions, so scores on DS-1000 are not directly comparable to scores on those benchmarks even for the same model.\n\n## Saturation and contamination\n\nThe paper's own 2022 ceiling, Codex-002 at 43.3%, is now a historical reference rather than a current one, and this page could not confirm a present-day top score from a maintained leaderboard, so saturation status is recorded as unknown rather than guessed. Contamination risk sits at medium: problems and reference solutions have been fully public since November 2022, and their Stack Overflow sources were public even earlier, so a model trained on recent web data could plausibly have encountered both the perturbed problem text and, for problems close to their original, something near its solution -- the perturbation strategy was designed to blunt simple answer memorisation, not to prevent the newer problem-and-solution pairs themselves from later appearing in training data.\n\n## How to run it\n\ninspect_evals implements DS-1000 as the `ds1000` task, pulling the dataset from Hugging Face (`xlangai/DS-1000`), executing generated code inside a Docker sandbox, and reporting pass@1 as the share of problems whose solution passes all unit tests; it supports filtering by library and by perturbation type. OpenCompass ships several `ds1000` configuration variants -- plain generation, completion-style generation, and \"service eval\" variants that call an external execution service -- which are not directly comparable if they differ in prompt format (completion vs. instruction-style) or in how strictly the surface-form constraints are enforced.\n\n## Reading the numbers\n\nA high DS-1000 score shows a model can write correct, idiomatic calls into common data-science libraries for realistic, StackOverflow-flavoured problems, which is a narrower and more practical skill than general algorithmic coding ability measured by benchmarks like HumanEval. Because the benchmark's own reference point (Codex-002 at 43.3%) is from 2022, a modern model's score should be read against other modern models rather than against that historical baseline. Check whether a reported number is pass@1 with the surface-form constraints enforced, since dropping that check make some solutions pass that the benchmark's authors intended to fail, and check which library subset a partial score covers, since performance can vary substantially between, for example, well-documented NumPy calls and less common TensorFlow-specific code.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ds1000",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "Problems and their reference solutions have been fully public since the November 2022 release, and the source Stack Overflow questions were themselves already public before that, so a model could plausibly have seen both the original questions and, for problems close to their source, something near the released solution. The perturbation strategy (surface, semantic and difficulty rewrites on top of an original) was designed to blunt simple memorisation of the original Stack Overflow answer, but does not remove the risk that the perturbed problem and its own solution have themselves now been seen during training on post-2022 web data.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-SA 4.0 (GitHub repository)",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 1000,
   "size_note": "1,000 problems across seven libraries: Pandas (291), NumPy (220), Matplotlib (155), Scikit-learn (115), SciPy (106), PyTorch (68) and TensorFlow (45). Problems derive from 451 unique Stack Overflow questions; of the 1,000 final problems, roughly 451 are close to their original source and the remainder were produced via three perturbation strategies: surface perturbation (152 problems: rewording or reformatting without changing the solution), semantic perturbation (235 problems: changing the actual requirement, such as asking for a minimum instead of a maximum), and difficult rewrites (162 problems: combining perturbations to increase difficulty). Human annotators found only a 1.8% false-discovery rate when checking whether Codex-002 solutions that passed the automated checks were genuinely correct.\n",
   "splits": "single set of 1,000 problems; no train/dev/test split",
   "url": "https://huggingface.co/datasets/xlangai/DS-1000"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice E"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "ds1000",
   "lm_eval": "",
   "opencompass": "ds1000",
   "other": ""
  },
  "id": "ds1000",
  "last_updated": "",
  "leaderboard_url": "https://ds1000-code-gen.github.io",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "DS-1000 tests whether a model can write short, realistic Python code that calls data-science libraries correctly, rather than solving abstract algorithmic puzzles. Each of the 1,000 problems asks for a snippet using one of seven libraries -- NumPy, Pandas, Matplotlib, Scikit-learn, SciPy, PyTorch or TensorFlow -- to do something a real practitioner asked how to do on Stack Overflow, such as reshaping a dataframe or configuring a specific plot. Problems are deliberately perturbed from their original Stack Overflow source (reworded, given a different but equally hard requirement, or made harder) specifically so a model cannot simply reproduce a memorised Stack Overflow answer verbatim and still pass.\n",
  "metric": {
   "baseline_note": "Scoring is multi-criteria: functional correctness runs the generated code against test cases (an average of 1.6 per problem) and compares outputs, using statistical tests for floating-point or randomised results, while a separate surface-form check rejects solutions that violate an explicit API or keyword constraint (for example, requiring a vectorised operation instead of an explicit loop; about 19.4% of problems carry such a constraint). The paper reports Codex-002, the strongest system it tested, at 43.3% accuracy in 2022; no random or human baseline applies to a code-generation task scored this way.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1 (execution-based, plus surface-form constraint checks)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "DS-1000",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2211.11501",
   "title": "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation",
   "url": "https://arxiv.org/abs/2211.11501",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Yuhang Lai",
    "Chengxi Li",
    "Yiming Wang",
    "Tianyi Zhang",
    "Ruiqi Zhong",
    "Luke Zettlemoyer",
    "Scott Wen-tau Yih",
    "Daniel Fried",
    "Sida Wang",
    "Tao Yu"
   ],
   "org": "University of Hong Kong, with co-authors at Stanford, UC Berkeley, the University of Washington, Meta AI and Carnegie Mellon University",
   "url": "https://github.com/xlang-ai/DS-1000"
  },
  "released": "2022-11",
  "repo_url": "https://github.com/xlang-ai/DS-1000",
  "saturation": {
   "as_of": "",
   "note": "The paper's own headline number, \"the current best public system (Codex-002) achieves 43.3% accuracy,\" is from 2022 and is now badly out of date given how far code-focused models have advanced since; this page could not read a maintained current leaderboard (the project page describes an \"Eval-Arena Leaderboard\" but did not render usable entries through this page's fetch) to confirm a present-day top score, so saturation status is left unestablished rather than inferred from general knowledge of newer models' likely performance.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation",
    "url": "https://arxiv.org/abs/2211.11501"
   },
   {
    "accessed": "2026-09-08",
    "title": "DS-1000 paper, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/abs/2211.11501"
   },
   {
    "accessed": "2026-09-08",
    "title": "xlang-ai/DS-1000 GitHub repository",
    "url": "https://github.com/xlang-ai/DS-1000"
   },
   {
    "accessed": "2026-09-08",
    "title": "DS-1000 project page",
    "url": "https://ds1000-code-gen.github.io"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals ds1000 task",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/ds1000"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ds1000 dataset configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/ds1000"
   }
  ],
  "status": "active",
  "subcategory": "data-science code generation grounded in real StackOverflow problems",
  "summary": "1,000 data-science coding problems across seven Python libraries, perturbed from real StackOverflow questions and checked by execution plus surface-form API constraints.",
  "tags": [
   "coding",
   "python",
   "data-science",
   "execution-based",
   "stackoverflow"
  ],
  "task_format": "Python code generation from a natural-language problem plus surrounding code context (either completion-style or insertion-style), one function or snippet per problem.\n"
 }
}