OpenCompass A/B cloze on CodeXGLUE ClozeTest-maxmin: given masked code and a docstring, choose whether the blank is max or min.
unassessed
| Category | coding |
|---|---|
| Subcategory | binary cloze over masked max/min in multilingual source code |
| Page status | active |
| Metric | accuracy (AccEvaluator after first_capital_postprocess) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2615 |
| Dataset licence | MIT (CodeXGLUE repository LICENSE); OpenCompass dataset files on the Hub were not readable here |
| Publisher | Microsoft (CodeXGLUE); OpenCompass wrap |
clozetest_maxmin is OpenCompass's generation wrap of Microsoft CodeXGLUE ClozeTest-maxmin. Each item is a function with one blank, a natural-language docstring, and a two-word answer set {max, min}. The model must decide which token belongs in the blank. CodeXGLUE built the items from CodeSearchNet validation and test functions in Ruby, JavaScript, Go, Python, Java, and PHP. The original CodeXGLUE task is masked-token classification. OpenCompass asks for the letter A (max) or B (min) in a chat turn instead of scoring an MLM head.
Zero-shot generation. OpenCompass prompt: code, docstring (nl_tokens), then "Please tell me what \"<mask>\" in the code should be replaced with and you must response to me only A or B." Gold is A if the answer file says max, else B. Config directory clozeTest_maxmin; dataset abbr maxmin; hashed config clozeTest_maxmin_gen_c205fb.py.
No model card in ModelSpec reports this benchmark yet.