Crowd-sourced, entry-level Python programming problems checked by unit tests; reported numbers vary widely because at least three differently-sized versions of the dataset are in circulation.
unassessed
| Category | coding |
|---|---|
| Subcategory | crowd-sourced, entry-level Python function generation |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 974 |
| Dataset licence | CC BY 4.0 (dataset, per the Hugging Face card); EvalPlus's separate MBPP+ repackaging is licensed Apache-2.0 |
| Publisher | Google Research |
MBPP gives a model a short natural-language description of a simple programming task -- for example, "write a function to find the shared elements from the given two lists" -- and asks it to produce a self-contained Python function that satisfies it. The original protocol also shows the model one of the three held-out test cases as a disambiguating hint. Problems were crowd-sourced from people with basic Python knowledge rather than handwritten by the paper's authors, so they skew toward short, everyday programming idioms (string and list manipulation, simple arithmetic, basic data-structure use) rather than algorithmic puzzles. It is a single-turn, text-to-code, Python-only task.
Given a one-to-two sentence task description (plus, in the original protocol, one example test case), generate a complete Python function; graded by executing the completion against a held-out set of unit tests (pass@k). Which problems, and how many tests each carries, differs by which released version of the dataset is used (see Dataset and licence).
No model card in ModelSpec reports this benchmark yet.