MBPP+

EvalPlus's stricter MBPP: the same crowd-sourced Python tasks, filtered and graded against about 35 times more tests so fragile completions fail.

Also known as: MBPP Plus, mbppplus, EvalPlus MBPP+

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategoryfunction-level Python generation with an expanded unit-test suite
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset size378
Dataset licenceApache-2.0
PublisherUniversity of Illinois Urbana-Champaign; Nanjing University

What it measures

MBPP+ keeps MBPP's short natural-language Python tasks and changes the grader. EvalPlus drops ill-formed items from the sanitized MBPP subset and replaces each problem's original three asserts with a much larger automatic test suite (the project documents about 35 times more tests than original MBPP). A completion that passes the light suite can still fail an edge case. The task remains single-turn, English prompt to Python function.

Task format

Generate a Python function from a short task description (OpenCompass's default config is 3-shot with [BEGIN]/[DONE] delimiters; EvalPlus's own CLI uses its package prompts). Graded pass@k against EvalPlus tests.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub