{
 "body": "\n## What it measures\n\nOpenCompass `clozeTest_maxmin` tests a narrow code-understanding choice. The model sees a function with a `<mask>` and a short natural-language description of what the code is for. It must say whether the missing token is `max` or `min`. That is a cloze over comparison operators, not general code generation.\n\nMicrosoft built ClozeTest-maxmin as one CodeXGLUE task. Blanks come from CodeSearchNet functions in six languages. ClozeTest-all, which is not this id, uses 930 candidate words. This id keeps only the two-word set. HELM and lm-eval do not register this OpenCompass spelling.\n\n## How it is scored\n\nOpenCompass uses `GenInferencer` with `ZeroRetriever` and `AccEvaluator`. A postprocessor keeps the first capital letter. The loader maps the answer file's `max`/`min` onto A/B. Chance is 50% if labels are balanced; the CodeXGLUE README does not state the max vs min split.\n\nThe CodeXGLUE README reports MLM accuracy (CodeBERT 85.66% overall). OpenCompass does not score an MLM head. Treat 85.66% as a related but different protocol.\n\n## Dataset and licence\n\nCodeXGLUE lists 2,615 examples: Python 1,264, Java 482, PHP 407, JavaScript 272, Go 152, Ruby 38. OpenCompass points at Hub paths `opencompass/clozeTest_maxmin` and `opencompass/clozeTest_maxmin_answers`. Those Hub endpoints returned HTTP 401 here, so this page does not re-count the JSON. The CodeXGLUE repository LICENSE is MIT. OpenCompass's own code is Apache-2.0; that is not automatically the dataset licence.\n\nAnswers travel in a second file joined on `idx` with a `<CODESPLIT>` delimiter. When `answer_path` is omitted, the loader stores an empty answer string.\n\n## Who publishes it\n\nCodeXGLUE (Shuai Lu, Daya Guo, Shuo Ren, and coauthors, Microsoft and collaborators) published the suite on 9 February 2021 (arXiv 2102.04664). The cloze data reuse CodeSearchNet (Husain et al.). OpenCompass added the A/B generation config under `configs/datasets/clozeTest_maxmin`. There is no separate cloze paper beyond CodeXGLUE and CodeBERT.\n\n## Lineage\n\nCodeSearchNet functions were turned into clozes for CodeBERT-style models, then folded into CodeXGLUE as ClozeTest-maxmin and ClozeTest-all. OpenCompass keeps the max/min subset and changes the interface from MLM to letter generation. This repository has no `codexglue` or `clozetest_all` page. [humaneval](humaneval.md) and [mbpp](mbpp.md) generate code; they do not fill a two-word blank.\n\n## Saturation and contamination\n\nCodeBERT already reached the mid-80s on the MLM form in 2021. Whether chat models still miss the OpenCompass A/B form is not established. Functions and answers have been public for years, so contamination risk is high for GitHub-trained models.\n\n## How to run it\n\nIn OpenCompass, import `clozeTest_maxmin_gen.py`, which re-exports `maxmin_datasets` from `clozeTest_maxmin_gen_c205fb.py`. The runnable abbreviation is `maxmin`, while the config folder and census id are `clozeTest_maxmin`. You need both the test JSON and the answer file.\n\nDo not report a CodeBERT MLM number as an OpenCompass `maxmin` score. Do not mix ClozeTest-all (930-word) results into this id.\n\n## Reading the numbers\n\nA high OpenCompass accuracy means the model usually emitted A or B in line with whether the blank was max or min. It does not mean the model can write the function, pass unit tests, or handle other masked identifiers. Python dominates the 2,615-item mix, so an overall number is not a six-language micro average unless the report says so. If a paper quotes 85.66%, it is almost certainly the CodeXGLUE MLM baseline, not this harness.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "clozetest_maxmin",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "clozeTest_maxmin",
   "CodeXGLUE ClozeTest-maxmin",
   "maxmin"
  ],
  "category": "coding",
  "contamination": {
   "note": "Functions come from CodeSearchNet and have been public in CodeXGLUE since 2021, with answers distributed beside the code. Models trained on GitHub or on CodeSearchNet can have seen the unmasked functions.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "code"
   ],
   "license": "MIT (CodeXGLUE repository LICENSE); OpenCompass dataset files on the Hub were not readable here",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": true,
   "size": 2615,
   "size_note": "CodeXGLUE ClozeTesting-maxmin README language counts: Ruby 38, JavaScript 272, Go 152, Python 1,264, Java 482, PHP 407, all 2,615. OpenCompass loads a JSON test_path (opencompass/clozeTest_maxmin) plus an answer file (opencompass/clozeTest_maxmin_answers) keyed by idx. The Hub copies were HTTP 401 from this session, so those 2,615 figures are from the CodeXGLUE README, not a re-count of the OpenCompass JSON.\n",
   "splits": "CodeXGLUE uses CodeSearchNet validation/test functions as a single cloze set; OpenCompass exposes a test_path plus a separate answer_path",
   "url": "https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/ClozeTesting-maxmin"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-031 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-031"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "clozeTest_maxmin",
   "other": ""
  },
  "id": "clozetest_maxmin",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "clozetest_maxmin is OpenCompass's generation wrap of Microsoft CodeXGLUE ClozeTest-maxmin. Each item is a function with one blank, a natural-language docstring, and a two-word answer set {max, min}. The model must decide which token belongs in the blank. CodeXGLUE built the items from CodeSearchNet validation and test functions in Ruby, JavaScript, Go, Python, Java, and PHP. The original CodeXGLUE task is masked-token classification. OpenCompass asks for the letter A (max) or B (min) in a chat turn instead of scoring an MLM head.\n",
  "metric": {
   "baseline_note": "Two options give a 50% chance rate. CodeXGLUE's README table is MLM accuracy (CodeBERT 85.66% overall, RoBERTa-base 62.45%), not OpenCompass A/B generation accuracy. Do not copy those rows onto an OpenCompass score.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (AccEvaluator after first_capital_postprocess)",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "ClozeTest-maxmin",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2102.04664",
   "title": "CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generation",
   "url": "https://arxiv.org/abs/2102.04664",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Shuai Lu",
    "Daya Guo",
    "Shuo Ren",
    "Junjie Huang",
    "Nan Duan"
   ],
   "org": "Microsoft (CodeXGLUE); OpenCompass wrap",
   "url": "https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/ClozeTesting-maxmin"
  },
  "released": "2021-02",
  "repo_url": "https://github.com/microsoft/CodeXGLUE/tree/main/Code-Code/ClozeTesting-maxmin",
  "saturation": {
   "as_of": "",
   "note": "CodeBERT's 85.66% is the CodeXGLUE MLM baseline on the same blanks, not an OpenCompass generation ceiling. No OpenCompass leaderboard figure for abbr maxmin was found during this research.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass clozeTest_maxmin configs",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/clozeTest_maxmin"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass hashed maxmin generation config",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/clozeTest_maxmin/clozeTest_maxmin_gen_c205fb.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass MaxminDataset loader",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/clozeTest_maxmin.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeXGLUE ClozeTest-maxmin README",
    "url": "https://raw.githubusercontent.com/microsoft/CodeXGLUE/main/Code-Code/ClozeTesting-maxmin/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeXGLUE MIT license",
    "url": "https://raw.githubusercontent.com/microsoft/CodeXGLUE/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "CodeXGLUE paper",
    "url": "https://arxiv.org/abs/2102.04664"
   }
  ],
  "status": "active",
  "subcategory": "binary cloze over masked max/min in multilingual source code",
  "summary": "OpenCompass A/B cloze on CodeXGLUE ClozeTest-maxmin: given masked code and a docstring, choose whether the blank is max or min.",
  "tags": [
   "code",
   "cloze",
   "codexglue",
   "opencompass"
  ],
  "task_format": "Zero-shot generation. OpenCompass prompt: code, docstring (nl_tokens), then \"Please tell me what \\\"<mask>\\\" in the code should be replaced with and you must response to me only A or B.\" Gold is A if the answer file says max, else B. Config directory clozeTest_maxmin; dataset abbr maxmin; hashed config clozeTest_maxmin_gen_c205fb.py.\n"
 }
}