OpenCompass's Chinese-instruction variant of MBPP: the same unit-tested Python tasks, prompted in Chinese rather than English.
unassessed
| Category | coding |
|---|---|
| Subcategory | entry-level Python function generation with Chinese-language instructions |
| Page status | active |
| Metric | pass@1 (OpenCompass MBPPEvaluator score, percent of tasks whose tests all pass) |
| Direction | higher_is_better |
| Unit | % |
| Publisher | OpenCompass Contributors |
mbpp_cn is OpenCompass's Chinese-language wrap of Mostly Basic Python Problems. The model still has to write a short Python function that passes hidden asserts, but the wrapper instruction and the three few-shot exemplars are written in Chinese. The current config loads a local JSONL at ./data/mbpp_cn/mbpp_cn.jsonl through the same MBPPDataset class used for English MBPP, so the {text} field is whatever that file stores — likely translated prompts, though the JSONL itself was not opened here. Unit tests in the few-shot block remain English assert statements. It is a single-turn, text-to-code task.
Three-shot generation. Each turn is a Chinese expert-programmer instruction, a task description, and English assert tests; the model continues after a [BEGIN] delimiter and is graded by executing the completion.
No model card in ModelSpec reports this benchmark yet.