{
 "body": "\n## What it measures\n\nMBPP+ is the EvalPlus re-test of Mostly Basic Python Problems. The model\nstill sees a short English task (for example, find shared list elements)\nand must write a Python function. What changes is the check: EvalPlus\nkeeps the sanitized MBPP prompts, drops broken items, and grades each\ncompletion against a generated test suite the project describes as about\n35 times larger than MBPP's original three asserts. A solution that was\n\"correct\" under the light suite can fail here.\n\n## How it is scored\n\nThe metric is pass@k, usually greedy pass@1. EvalPlus scores the same\ncompletions two ways: original asserts (`mbpp`) and plus tests (`mbpp+`).\nThose two numbers are not interchangeable. OpenCompass's `mbpp_plus`\nconfig calls `MBPPEvaluator(metric='MBPPPlus')`, which imports EvalPlus\nand evaluates `dataset='mbpp'` samples against plus tests. The default\nOpenCompass prompt is 3-shot with the same three exemplars as English\nMBPP; `mbpp_plus_0shot_new_gen.py` is a separate zero-shot template.\n\n## Dataset and licence\n\n`evalplus/mbppplus` on Hugging Face has one test split of 378 examples,\nlicence Apache-2.0, last updated 2024-04-17 on the Hub API. EvalPlus's\nown news says MBPP+ started at 399 tasks and lost broken items down to\n378 in the v0.2.0 dataset refresh on 17 April 2024. Prompts are English;\nthe programming language is Python. Tests are public.\n\n## Who publishes it\n\nJiawei Liu, Chunqiu Steven Xia, Yuyao Wang and Lingming Zhang\n(UIUC; Wang at Nanjing University) introduced the EvalPlus test-augmentation\nmethod in the NeurIPS 2023 paper posted May 2023 (arXiv 2305.01210).\nMBPP+ as a released dataset appears in the EvalPlus repository and Hub\nin January 2024, with the 378-task v0.2.0 cut in April 2024. The\nleaderboard is evalplus.github.io.\n\n## Lineage\n\nPredecessor: [MBPP](mbpp.md). Sibling under the same method:\n[HumanEval+](humaneval_plus.md). [MBPP Pro](mbpp_pro.md) is a later\nself-invoking task that happens to use 378 pairs (the Hugging Face\nCodeEval-Pro/mbpp-pro row count). It does not replace these plus tests.\nOpenCompass ships `mbpp_plus` as its own dataset abbreviation.\n\n## Saturation and contamination\n\nOn EvalPlus `results.json` (2026-09-08), o1-preview (Sept 2024) is 80.2\npass@1 on mbpp+ and 95.5 on mbpp. That 15-point drop is the point of the\nbenchmark. The file does not include 2025\u20132026 frontier models, so 80.2\nis not a current ceiling. Contamination risk is high: MBPP prompts and\nsolutions have been public since 2021, and the plus tests are public too.\n\n## How to run it\n\n`pip` package evalplus: `evalplus.evaluate --dataset mbpp --greedy` (and\nbackends documented in the README). OpenCompass: configs under\n`opencompass/configs/datasets/mbpp_plus/`, abbreviation `mbpp_plus`.\nAlways record whether a published \"MBPP\" number used plus tests, original\ntests, 3-shot OpenCompass, or zero-shot EvalPlus prompts.\n\n## Reading the numbers\n\nA high MBPP+ score is stronger evidence of correct Python than the same\nscore on plain MBPP, because it survived a much larger test suite. The\ngap between the two columns is itself a result: a large gap means the\nmodel is skating through weak asserts. It is still short, self-contained\nPython, with public prompts, and it is not software-engineering in the\nlarge. Pair it with [LiveCodeBench](live_code_bench.md) or [MBPP Pro](mbpp_pro.md)\nwhen you care about contamination or composition.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mbpp_plus",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "MBPP Plus",
   "mbppplus",
   "EvalPlus MBPP+"
  ],
  "category": "coding",
  "contamination": {
   "note": "Prompts and canonical solutions come from MBPP, public since August 2021. Extra tests catch fragile code; they do not hide the problem text. EvalPlus's tests and oracles have themselves been public since 2024.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 378,
   "size_note": "Hugging Face evalplus/mbppplus, default config, single test split, 378 rows (Hub API and datasets-server, accessed 2026-09-08). EvalPlus news records 399 tasks at first MBPP+ release, reduced to 378 after removing broken tasks on 2024-04-17 (dataset labelled v0.2.0). This repository's mbpp page independently notes that lineage.\n",
   "splits": "single 378-row test split; no train or validation split",
   "url": "https://huggingface.co/datasets/evalplus/mbppplus"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "mbpp_plus",
   "other": "Reference: `evalplus.evaluate --dataset mbpp` (plus tests are the default mbpp+ column). OpenCompass abbr mbpp_plus, MBPPPlusDataset + MBPPEvaluator(metric='MBPPPlus') calling evalplus; default mbpp_plus_gen_0b836a.py is 3-shot, mbpp_plus_0shot_new_gen.py is a zero-shot prompt variant. bigcode-evaluation-harness wraps the dataset as well."
  },
  "id": "mbpp_plus",
  "last_updated": "2024-04",
  "leaderboard_url": "https://evalplus.github.io/leaderboard.html",
  "lineage": {
   "family": "",
   "predecessor": "mbpp",
   "successors": [],
   "variants": [
    "humaneval_plus"
   ]
  },
  "measures": "MBPP+ keeps MBPP's short natural-language Python tasks and changes the grader. EvalPlus drops ill-formed items from the sanitized MBPP subset and replaces each problem's original three asserts with a much larger automatic test suite (the project documents about 35 times more tests than original MBPP). A completion that passes the light suite can still fail an edge case. The task remains single-turn, English prompt to Python function.\n",
  "metric": {
   "baseline_note": "Same unbiased pass@k estimator as HumanEval+/MBPP, computed on the expanded tests. EvalPlus also reports a base \"mbpp\" column on the same completions using the original asserts. No human baseline is published. Random chance of passing a large hidden suite is effectively zero.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "MBPP+",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2305.01210",
   "title": "Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation",
   "url": "https://arxiv.org/abs/2305.01210",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Jiawei Liu",
    "Chunqiu Steven Xia",
    "Yuyao Wang",
    "Lingming Zhang"
   ],
   "org": "University of Illinois Urbana-Champaign; Nanjing University",
   "url": "https://github.com/evalplus/evalplus"
  },
  "released": "2024-01",
  "repo_url": "https://github.com/evalplus/evalplus",
  "saturation": {
   "as_of": "2024-09",
   "note": "EvalPlus results.json (fetched 2026-09-08) has a top mbpp+ pass@1 of 80.2 for OpenAI o1-preview (September 2024), with o1-mini at 78.8. The same o1-preview run is 95.5 on the original-assert \"mbpp\" column. Newest named entries on that file are around late 2024, so the 80.2 figure is dated. The gap versus the light suite is why this is \"watch\" rather than saturated.\n",
   "status": "watch",
   "top_score": 80.2
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "EvalPlus paper (NeurIPS 2023)",
    "url": "https://arxiv.org/abs/2305.01210"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/evalplus README (MBPP+ 35x tests; 399 to 378 on 2024-04-17)",
    "url": "https://github.com/evalplus/evalplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/mbppplus Hub API (378-row test split, Apache-2.0)",
    "url": "https://huggingface.co/api/datasets/evalplus/mbppplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "evalplus/mbppplus splits",
    "url": "https://datasets-server.huggingface.co/splits?dataset=evalplus/mbppplus"
   },
   {
    "accessed": "2026-09-08",
    "title": "EvalPlus leaderboard results.json",
    "url": "https://evalplus.github.io/results.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass mbpp_plus 3-shot config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/mbpp_plus/mbpp_plus_gen_0b836a.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MBPPPlusDataset and MBPPPlus evaluator",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/mbpp.py"
   }
  ],
  "status": "active",
  "subcategory": "function-level Python generation with an expanded unit-test suite",
  "summary": "EvalPlus's stricter MBPP: the same crowd-sourced Python tasks, filtered and graded against about 35 times more tests so fragile completions fail.",
  "tags": [
   "code-generation",
   "python",
   "pass-at-k",
   "evalplus",
   "unit-tests"
  ],
  "task_format": "Generate a Python function from a short task description (OpenCompass's default config is 3-shot with [BEGIN]/[DONE] delimiters; EvalPlus's own CLI uses its package prompts). Graded pass@k against EvalPlus tests.\n"
 }
}