Microsoft CodeXGLUE is a 10-task, 14-dataset suite for code understanding and generation; EleutherAI lm-eval currently ships only the code-to-text group.
unassessed
| Category | coding |
|---|---|
| Subcategory | Microsoft 10-task code understanding and generation suite; lm-eval ships code-to-text |
| Page status | active |
| Metric | smoothed BLEU-4 (lm-eval code2text); suite-wide metrics vary by task |
| Direction | higher_is_better |
| Dataset licence | MIT (repository code); C-UDA (datasets, per README) |
| Publisher | Microsoft Research Asia, Developer Division, and Bing |
CodeXGLUE (General Language Understanding Evaluation for CODE) is Microsoft's 2021 suite of 10 code-intelligence tasks on 14 datasets. The four scenarios are code-code (clone detection, defect detection, cloze, completion, repair, translation), text-code (search and text-to-code), code-text (summarization), and text-text (documentation translation). EleutherAI lm-evaluation-harness implements only the code-to-text group under lm_eval/tasks/code_x_glue: generate a natural-language docstring from tokenized code in Go, Java, JavaScript, PHP, Python, or Ruby. That wrap is CodeSearchNet-derived summarization scored with smoothed BLEU-4, not the whole suite.
lm-eval code2text: generate_until on Hub datasets CM/codexglue_code2text_*. doc_to_text joins code_tokens; doc_to_target joins docstring_tokens. Beam 10, max_gen_toks 128, stop at </s>. Official CodeXGLUE tasks use task-specific metrics (accuracy, MAP, CodeBLEU, BLEU, EM) and often hold out test labels for email submission.
No model card in ModelSpec reports this benchmark yet.