OpenCompass wrap of CodeXGLUE's PY150 line-level completion: given a Python prefix, generate the next line and score it with BLEU.
unassessed
| Category | coding |
|---|---|
| Subcategory | next-line Python completion from CodeXGLUE PY150 |
| Page status | active |
| Metric | BLEU (OpenCompass BleuEvaluator) |
| Direction | higher_is_better |
| Dataset size | 10000 |
| Dataset licence | C-UDA (CodeXGLUE datasets); MIT (CodeXGLUE repository code) |
| Publisher | Microsoft Research Asia, Developer Division, and Bing (CodeXGLUE); OpenCompass wrap |
py150 in OpenCompass asks a model to write the next line of Python given the lines that precede it. The underlying files come from Raychev, Bielik, and Vechev's PY150 corpus of 150,000 Python files, as packaged by Microsoft CodeXGLUE for line-level completion. CodeXGLUE built the line-level test by cutting a file at a random point and taking tokens until the next end-of-line marker. OpenCompass then turns that prefix/target pair into a chat prompt ("write down what the next line of code is") and scores the generation with BLEU rather than CodeXGLUE's official exact match and edit similarity.
Zero-shot generation. OpenCompass loads data/py150/test.json, strips tagged literals, joins <EOL>-separated tokens into a multiline prefix, and asks for a single next line. Gold is the gt field.
No model card in ModelSpec reports this benchmark yet.