{
 "body": "\n## What it measures\n\nMBPP (Mostly Basic Python Problems) gives a model a short natural-language description of a simple\nprogramming task -- for example, \"write a function to find the shared elements from the given two\nlists\" -- and asks it to produce a self-contained Python function that satisfies it. The original\nevaluation protocol also shows the model one of the problem's three held-out test cases as a\ndisambiguating hint before it answers. Unlike HumanEval, whose problems were handwritten by its\nauthors, MBPP's problems were crowd-sourced from people with basic Python knowledge, so they skew\ntoward short, everyday programming idioms -- string and list manipulation, simple arithmetic, basic\ndata-structure use -- rather than algorithmic puzzles. It is a single-turn, text-to-code,\nPython-only task.\n\n## How it is scored\n\nA completion is graded by executing it against a held-out set of unit tests; a problem counts as\nsolved only if all tests pass. The paper's own headline metric is few-shot pass@1, sampled with a\nfixed 3-shot prompt (task IDs 2, 3 and 4) and estimated across multiple samples; most current\nreporters instead run one greedy completion and call it pass@1, which is cheaper but not numerically\nidentical. Because at least three differently sized, differently filtered versions of the dataset\nare in active use -- the original 974-problem full set, a smaller hand-verified \"sanitized\" subset,\nand EvalPlus's further-filtered, test-augmented MBPP+ -- two \"MBPP pass@1\" numbers can disagree by\ndouble digits without either being wrong, simply because they were computed against different\nproblems with different tests.\n\n## Dataset and licence\n\nThe original release contains 974 crowd-sourced problems, each with a task description, a\nreference solution and 3 unit tests, split by the authors into task IDs 1-10 (few-shot prompt\npool), 11-510 (500-problem test split), 511-600 (90-problem validation split) and 601-974\n(374-problem training split) -- confirmed against the current Hugging Face mirror's \"full\" config.\nThe authors separately hand-inspected, edited and pruned a subset for clarity, describing it in the\npaper as \"426 hand-verified questions\"; the mirror's \"sanitized\" config instead totals 427, a small\ndiscrepancy this page reports rather than resolves. EvalPlus later built MBPP+ on top of the\nsanitized subset by dropping further ill-formed problems (399 at its January 2024 release, reduced\nto 378 after an April 2024 fix) and adding roughly 35x more automatically generated tests per\nproblem, released separately as evalplus/mbppplus. The underlying dataset carries a CC BY 4.0\nlicence; MBPP+'s repackaging is separately licensed Apache-2.0. All text is English; the only\nprogramming language covered is Python.\n\n## Who publishes it\n\nMBPP was introduced by Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk\nMichalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le and Charles Sutton, all\nat Google Research, in \"Program Synthesis with Large Language Models,\" posted to arXiv in August\n2021 alongside the paper's other benchmark, MathQA-Python. Google Research maintains the reference\ndataset on GitHub. Since 2024 it has been most visibly extended and re-reported by EvalPlus\n(evalplus.github.io), an independent project that also produced HumanEval+ and maintains a\ncommunity leaderboard covering both original- and plus-style scores.\n\n## Lineage\n\nMBPP has no formal predecessor. Its most consequential documented descendant is EvalPlus's MBPP+,\nwhich keeps the same problem descriptions but multiplies the test suite roughly 35-fold to catch\nsolutions that pass the original three tests without being genuinely correct -- this repository\ndoes not yet have a separate mbpp_plus page. MultiPL-E (multipl_e in this repository) mechanically\ntranslates both HumanEval's and MBPP's problems into 18+ other programming languages, so MBPP\nunderlies part of that family too. The LiveCodeBench paper (arXiv 2403.07974) names MBPP directly,\nalongside HumanEval, as an example of an existing benchmark \"no longer sufficient\" for evaluating\ncurrent models, motivating LiveCodeBench's (live_code_bench) contamination-resistant, continuously\nrefreshed problem collection -- a response to MBPP's limitations rather than a formal replacement.\n\n## Saturation and contamination\n\nOn the EvalPlus leaderboard's underlying results data (fetched 2026-09-08), the top model on the\nlightly-tested \"mbpp\" column, OpenAI's o1-preview (September 2024), scored 95.5% pass@1 -- near the\nceiling -- while the same model scored 80.2% on the stricter \"mbpp+\" column, a 15-point gap showing\nhow much of that apparent saturation comes from a weak original test suite rather than verified\ncorrectness. That leaderboard's newest entries date to around September-November 2024, so neither\nfigure reflects current frontier models. Contamination risk is high: both problem sets, with\nreference solutions, have been public since August 2021 and are widely re-hosted, including inside\ninstruction-tuning corpora that bundle text-to-code pairs; MBPP+ reuses the same problem text and so\ncarries the same exposure.\n\n## How to run it\n\nThree widely used harnesses score three different problem sets under the same \"mbpp\" name.\nlm-evaluation-harness's `mbpp` task reads the \"full\" config's 500-item test split, 3-shot, using the\nsame prompt template and [BEGIN]/[DONE] delimiters as the original paper. inspect_evals's `mbpp`\ntask instead reads the \"sanitized\" split and runs 5 sampling epochs per problem. OpenCompass ships\nits own `mbpp` dataset configuration. EvalPlus is a separate, pip-installable package\n(`evalplus.evaluate --dataset mbpp`) that runs the 378-problem filtered set and can additionally\nscore the same completions against the larger MBPP+ test suite. Because none of these read the same\nset of problems, a score is only comparable to another score computed with the same harness and\ndataset config.\n\n## Reading the numbers\n\nA high MBPP pass@1 shows a model can turn an everyday, precisely stated task description into a\nshort working Python function -- useful signal for basic coding fluency, but not evidence of the\nmulti-file, tool-using or debugging skills real software engineering requires. Because at least\nthree differently sized versions of \"MBPP\" circulate, a single reported percentage is close to\nmeaningless without knowing which dataset and harness produced it; the gap between plain \"mbpp\" and\n\"mbpp+\" scores for the same model is itself informative, since a large gap suggests the model is\npassing weak tests rather than writing robust code. Given the dataset's age and public solutions,\ntreat a high score as necessary but not sufficient, and check a contamination-resistant benchmark\nsuch as LiveCodeBench alongside it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mbpp",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Mostly Basic Python Problems"
  ],
  "category": "coding",
  "contamination": {
   "note": "Both the crowd-sourced full dataset and the hand-verified sanitized subset have been public, with reference solutions included, since August 2021 -- long enough to appear in the training data of most models trained on a broad web or code crawl since, and the dataset is widely re-hosted (Hugging Face, Papers with Code, numerous instruction-tuning corpora that bundle text-to-code pairs). EvalPlus's MBPP+ reuses the same underlying problem descriptions and only adds new test cases, so it carries the same exposure. The LiveCodeBench paper (arXiv 2403.07974) names MBPP directly, alongside HumanEval, as an example of a benchmark \"no longer sufficient\" for evaluating current models for this reason.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0 (dataset, per the Hugging Face card); EvalPlus's separate MBPP+ repackaging is licensed Apache-2.0",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 974,
   "size_note": "The original release contains 974 crowd-sourced problems (task description, reference solution and 3 unit tests each), split by the authors' own repository into task IDs 1-10 (few-shot prompt pool), 11-510 (500-problem test split, used for scoring), 511-600 (90-problem validation split) and 601-974 (374-problem training/fine-tuning split) -- confirmed against the google-research-datasets/mbpp Hugging Face mirror, whose \"full\" config splits match those counts exactly (train 374, test 500, validation 90, prompt 10). The paper's authors separately hand-inspected, edited and pruned a subset for clarity, reporting \"426 hand-verified questions\" in the paper itself; the \"sanitized\" config of the same Hugging Face mirror instead totals 427 (120 train, 257 test, 43 validation, 7 prompt) -- a small, unexplained difference between the paper's stated count and the released sanitized split that this page reports rather than resolves. EvalPlus built a further-filtered, test-augmented version on top of the sanitized subset: 399 problems at its first release (2024-01), reduced to 378 after removing broken tasks in the v0.2.0 update (2024-04-17), each carrying roughly 35x more automatically generated test cases than the original 3 -- released as MBPP+ (evalplus/mbppplus on Hugging Face, 378 rows in its single \"test\" split, confirmed via the datasets-server API). None of these three counts (974, 427/426, 378) is \"the\" size of MBPP; which one a reported score used should be checked before comparing two numbers.\n",
   "splits": "\"full\" config: train 374, test 500, validation 90, prompt 10 (974 total). \"sanitized\" config: train 120, test 257, validation 43, prompt 7 (427 total). EvalPlus MBPP+: a single 378-row test split with no train/validation division.\n",
   "url": "https://huggingface.co/datasets/google-research-datasets/mbpp"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "mbpp",
   "lm_eval": "mbpp",
   "opencompass": "mbpp",
   "other": "EvalPlus (pip package `evalplus`, evalplus.github.io): `evalplus.evaluate --dataset mbpp` runs the 378-problem filtered set with its original tests; the same tool also computes the \"mbpp+\" score by checking the same completions against the ~35x-larger generated test suite. Confirmed directly: lm-evaluation-harness's `mbpp` task reads the \"full\" config's 500-item test split, 3-shot, with the same prompt template (\"You are an expert Python programmer...\") and [BEGIN]/ [DONE] delimiters as the original paper; inspect_evals's `mbpp` task instead reads the \"sanitized\" split and runs 5 epochs per problem. These are three different problem sets under one task name."
  },
  "id": "mbpp",
  "last_updated": "2024-04",
  "leaderboard_url": "https://evalplus.github.io/leaderboard.html",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "live_code_bench"
   ],
   "variants": [
    "mbpp_plus",
    "multipl_e"
   ]
  },
  "measures": "MBPP gives a model a short natural-language description of a simple programming task -- for example, \"write a function to find the shared elements from the given two lists\" -- and asks it to produce a self-contained Python function that satisfies it. The original protocol also shows the model one of the three held-out test cases as a disambiguating hint. Problems were crowd-sourced from people with basic Python knowledge rather than handwritten by the paper's authors, so they skew toward short, everyday programming idioms (string and list manipulation, simple arithmetic, basic data-structure use) rather than algorithmic puzzles. It is a single-turn, text-to-code, Python-only task.\n",
  "metric": {
   "baseline_note": "The original paper's headline number is few-shot pass@1 with 3-shot prompting (task IDs 2, 3 and 4 as the fixed prompt exemplars, per the reference repository), estimated from sampling rather than a single greedy completion. Most current reporters instead run a single greedy completion and call it pass@1, which is cheaper but not numerically identical to the paper's sampling-based estimate. No random-guess or human baseline is established by the paper or repository.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "MBPP (Mostly Basic Python Problems)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2108.07732",
   "title": "Program Synthesis with Large Language Models",
   "url": "https://arxiv.org/abs/2108.07732",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Jacob Austin",
    "Augustus Odena",
    "Maxwell Nye",
    "Maarten Bosma",
    "Henryk Michalewski",
    "David Dohan",
    "Ellen Jiang",
    "Carrie Cai",
    "Michael Terry",
    "Quoc Le",
    "Charles Sutton"
   ],
   "org": "Google Research",
   "url": "https://github.com/google-research/google-research/tree/master/mbpp"
  },
  "released": "2021-08",
  "repo_url": "https://github.com/google-research/google-research/tree/master/mbpp",
  "saturation": {
   "as_of": "2024-09",
   "note": "On the EvalPlus leaderboard's results.json (fetched 2026-09-08), the top model on the lightly-tested \"mbpp\" column was OpenAI's o1-preview (September 2024) at 95.5% pass@1 -- near the ceiling -- while the same model scored 80.2% on the far stricter \"mbpp+\" column, a 15-point gap that shows how much of the apparent saturation comes from a weak original 3-test suite rather than genuine correctness. The leaderboard's most recent entries date to around September-November 2024; no scores for newer 2025-2026 frontier models were found there, so this figure should not be read as a current state of the art. Given the wide, unresolved spread between mbpp and mbpp+ scores, and the leaderboard's apparent staleness, this is graded \"watch\" rather than \"saturated.\"\n",
   "status": "watch",
   "top_score": 95.5
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Program Synthesis with Large Language Models",
    "url": "https://arxiv.org/abs/2108.07732"
   },
   {
    "accessed": "2026-09-08",
    "title": "google-research/google-research: mbpp (dataset README, evaluation split definitions)",
    "url": "https://github.com/google-research/google-research/tree/master/mbpp"
   },
   {
    "accessed": "2026-09-08",
    "title": "google-research-datasets/mbpp dataset card and metadata, Hugging Face",
    "url": "https://huggingface.co/datasets/google-research-datasets/mbpp"
   },
   {
    "accessed": "2026-09-08",
    "title": "google-research-datasets/mbpp, Hugging Face Hub API",
    "url": "https://huggingface.co/api/datasets/google-research-datasets/mbpp"
   },
   {
    "accessed": "2026-09-08",
    "title": "google-research-datasets/mbpp split sizes, Hugging Face datasets-server",
    "url": "https://datasets-server.huggingface.co/splits?dataset=google-research-datasets/mbpp"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/evalplus repository (MBPP+ release notes and changelog)",
    "url": "https://github.com/evalplus/evalplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/mbppplus, Hugging Face Hub API (378-row test split, Apache-2.0)",
    "url": "https://huggingface.co/api/datasets/evalplus/mbppplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (EvalPlus)",
    "url": "https://arxiv.org/abs/2305.01210"
   },
   {
    "accessed": "2026-09-08",
    "title": "EvalPlus Leaderboard",
    "url": "https://evalplus.github.io/leaderboard.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: mbpp task config",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mbpp/mbpp.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals: mbpp task implementation",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/mbpp"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass dataset configs (includes mbpp)",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/mbpp"
   },
   {
    "accessed": "2026-09-08",
    "title": "LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code",
    "url": "https://arxiv.org/abs/2403.07974"
   }
  ],
  "status": "active",
  "subcategory": "crowd-sourced, entry-level Python function generation",
  "summary": "Crowd-sourced, entry-level Python programming problems checked by unit tests; reported numbers vary widely because at least three differently-sized versions of the dataset are in circulation.",
  "tags": [
   "code-generation",
   "python",
   "pass-at-k",
   "crowd-sourced",
   "unit-tests"
  ],
  "task_format": "Given a one-to-two sentence task description (plus, in the original protocol, one example test case), generate a complete Python function; graded by executing the completion against a held-out set of unit tests (pass@k). Which problems, and how many tests each carries, differs by which released version of the dataset is used (see Dataset and licence).\n"
 }
}