{
 "body": "\n## What it measures\n\nThis OpenCompass task measures next-line Python completion. The model sees a prefix of a real file and must emit the following line, not the next token and not a full function. The skill is local syntactic continuation: a signature, a loop header, or a call with arguments, given hundreds of tokens of file context.\n\nThe files are from PY150, a 2016 corpus of 150,000 Python files used in CodeXGLUE's code-completion track. OpenCompass does not run CodeXGLUE's token-level language-modelling setup. It only wraps the line-level test JSON.\n\n## How it is scored\n\nOpenCompass uses `BleuEvaluator` on the generated line against `gt`. CodeXGLUE's own line-completion evaluator reports exact match and edit similarity after mapping normalised literals such as `<NUM_LIT:1>` back to source text. Those two protocols are not interchangeable.\n\nThe hashed config is zero-shot (`ZeroRetriever`) with a chat template that forbids extra commentary. A BLEU run is only comparable to another OpenCompass `py150` run that used the same post-processing. The unused `first_capital_postprocess` import in the config is not applied.\n\n## Dataset and licence\n\nCodeXGLUE states the PY150 line-completion test has 10,000 examples. OpenCompass reads `data/py150/test.json` locally, one JSON object per line, with `input` and `gt` fields. The loader replaces `<NUM_LIT>`, `<STR_LIT>`, and tagged literals, then joins `<EOL>` tokens with newlines.\n\nCodeXGLUE datasets are published under the Computational Use of Data Agreement (C-UDA). Repository code is MIT. Token-level PY150 is 95,000 train / 5,000 dev / 50,000 test files; that is the pretraining split for CodeGPT, not the 10,000-row line test.\n\n## Who publishes it\n\nMicrosoft Research Asia, Developer Division, and Bing released CodeXGLUE in 2021 (arXiv 2102.04664). The PY150 files themselves are from Raychev, Bielik, and Vechev, OOPSLA 2016. OpenCompass maintains the chat wrap under abbr `py150`. The CodeXGLUE site hosts a completion leaderboard for the official EM/edit-similarity protocol, not for OpenCompass BLEU.\n\n## Lineage\n\nThis is one CodeXGLUE code-code task, not the whole [code_x_glue](code_x_glue.md) suite. lm-eval's CodeXGLUE directory only ships code-to-text summarisation, so this id is not a harness spelling of that page.\n\n[clozetest_maxmin](clozetest_maxmin.md) is a different CodeXGLUE wrap: binary max/min cloze on CodeSearchNet, not PY150 completion. Token-level PY150 language modelling is the sibling CodeXGLUE task and has no page here.\n\n## Saturation and contamination\n\nCodeGPT-adapted's 42.37 exact match is a 2021 baseline on the official metric. Current chat-model BLEU on OpenCompass `py150` is not established. Because the source is public GitHub Python from 2016, treat leaked-file memorisation as likely. A high BLEU on a unique next line can still be copy-from-training rather than completion skill.\n\n## How to run it\n\nUse OpenCompass with `opencompass/configs/datasets/py150/py150_gen.py`, which loads `py150_gen_38b13d.py`. Place CodeXGLUE's line-completion `test.json` at `data/py150/test.json`. The dataset class is `Py150Dataset`.\n\nDo not report CodeXGLUE EM next to an OpenCompass BLEU number. Do not mix Java Corpus line completion (3,000 examples, different language) into this id.\n\n## Reading the numbers\n\nA strong OpenCompass BLEU means the generated line overlaps the gold n-grams. It does not mean the line is executable or that the model could finish the rest of the file. Compare against [code_x_glue](code_x_glue.md) summarisation or [clozetest_maxmin](clozetest_maxmin.md) only as related CodeXGLUE family context, not as the same score. Prefer CodeXGLUE EM when you need the official completion table.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "py150",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "py150",
   "CodeXGLUE PY150 line completion"
  ],
  "category": "coding",
  "contamination": {
   "note": "PY150 is a 2016 public GitHub Python dump. CodeXGLUE's line-completion test is public JSON. Models trained on GitHub Python are likely to have seen the underlying files even if they have not seen this exact cut.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "Python"
   ],
   "license": "C-UDA (CodeXGLUE datasets); MIT (CodeXGLUE repository code)",
   "modalities": [
    "code"
   ],
   "public_test_set": true,
   "size": 10000,
   "size_note": "CodeXGLUE CodeCompletion-line README: PY150 line-completion test has 10,000 examples, mean 477.81 input tokens and 6.61 output tokens. OpenCompass Py150Dataset reads a local data/py150/test.json with one JSON object per line (input, gt), which is that test file after literal normalisation. The original token-level PY150 split is 95,000 / 5,000 / 50,000 files; that file count is not the OpenCompass eval size.\n",
   "splits": "OpenCompass exposes a single test.json; CodeXGLUE line-completion is test-only (10,000), trained from the token-level PY150 split",
   "url": "https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-line"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-067 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-067"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "py150",
   "other": "Config opencompass/configs/datasets/py150/py150_gen.py re-exports py150_gen_38b13d.py. Dataset class opencompass.datasets.Py150Dataset. Path data/py150/test.json (local_mode). Not in lm-eval; CodeXGLUE's official evaluator is exact match plus edit similarity.\n"
  },
  "id": "py150",
  "last_updated": "2021-07",
  "leaderboard_url": "https://microsoft.github.io/CodeXGLUE/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "py150 in OpenCompass asks a model to write the next line of Python given the lines that precede it. The underlying files come from Raychev, Bielik, and Vechev's PY150 corpus of 150,000 Python files, as packaged by Microsoft CodeXGLUE for line-level completion. CodeXGLUE built the line-level test by cutting a file at a random point and taking tokens until the next end-of-line marker. OpenCompass then turns that prefix/target pair into a chat prompt (\"write down what the next line of code is\") and scores the generation with BLEU rather than CodeXGLUE's official exact match and edit similarity.\n",
  "metric": {
   "baseline_note": "CodeXGLUE's published line-completion table is exact match and edit similarity (CodeGPT-adapted 42.37 EM / 71.59 edit similarity on PY150). OpenCompass's hashed config uses BleuEvaluator only. Do not copy an EM number onto an OpenCompass BLEU run.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "BLEU (OpenCompass BleuEvaluator)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "PY150 (OpenCompass line completion)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2102.04664",
   "title": "CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation",
   "url": "https://arxiv.org/abs/2102.04664",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Shuai Lu",
    "Daya Guo",
    "Shuo Ren",
    "Junjie Huang",
    "Alexey Svyatkovskiy",
    "Veselin Raychev",
    "Pavol Bielik",
    "Martin Vechev"
   ],
   "org": "Microsoft Research Asia, Developer Division, and Bing (CodeXGLUE); OpenCompass wrap",
   "url": "https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/CodeCompletion-line"
  },
  "released": "2021-02",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/py150",
  "saturation": {
   "as_of": "",
   "note": "CodeXGLUE's CodeGPT-adapted line-level EM of 42.37 is a 2021 encoder-decoder baseline, not an OpenCompass BLEU ceiling. No OpenCompass leaderboard figure for abbr py150 was found.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass configs/datasets/py150 directory",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/py150"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass py150_gen.py (re-export of hashed config)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/py150/py150_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass py150_gen_38b13d.py (prompt, BleuEvaluator, abbr py150)",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/py150/py150_gen_38b13d.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass Py150Dataset loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/py150.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeXGLUE CodeCompletion-line README (10,000 PY150 test rows, EM/edit sim)",
    "url": "https://raw.githubusercontent.com/microsoft/CodeXGLUE/main/Code-Code/CodeCompletion-line/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeXGLUE CodeCompletion-token README (PY150 file splits, C-UDA literals)",
    "url": "https://raw.githubusercontent.com/microsoft/CodeXGLUE/main/Code-Code/CodeCompletion-token/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeXGLUE root README (MIT code, C-UDA datasets)",
    "url": "https://raw.githubusercontent.com/microsoft/CodeXGLUE/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv abs CodeXGLUE (2102.04664)",
    "url": "https://arxiv.org/abs/2102.04664"
   }
  ],
  "status": "active",
  "subcategory": "next-line Python completion from CodeXGLUE PY150",
  "summary": "OpenCompass wrap of CodeXGLUE's PY150 line-level completion: given a Python prefix, generate the next line and score it with BLEU.\n",
  "tags": [
   "coding",
   "code-completion",
   "python",
   "opencompass"
  ],
  "task_format": "Zero-shot generation. OpenCompass loads data/py150/test.json, strips tagged literals, joins <EOL>-separated tokens into a multiline prefix, and asks for a single next line. Gold is the gt field.\n"
 }
}