{
 "body": "\n## What it measures\n\nMultiPL-E asks whether a code model's ability on Python function-completion problems generalises to\nother programming languages. It starts from two established Python benchmarks, HumanEval (164\nhand-written problems, each a function signature, docstring and tests) and MBPP (a larger pool of\ncrowd-written \"basic Python\" problems), and mechanically re-expresses each one in a target language: the\nsignature, the docstring, and the tests are all translated by a small per-language compiler, while the\nunderlying algorithm the model must implement stays the same.\n\nThe model sees only the translated prompt in the target language and must complete the function body. It\nis not asked to translate code itself, use language-specific libraries, or write anything beyond what a\nsingle self-contained function requires, so a low score is more likely to reflect syntax and standard-\nlibrary unfamiliarity than a lack of algorithmic reasoning.\n\n## How it is scored\n\nEach generated completion is dropped into a small per-language test harness, compiled or interpreted with\nthat language's real toolchain inside a container, and checked against the translated unit tests. The\nheadline metric is pass@1, and the reference implementation also supports pass@10 and pass@100 by\nsampling more completions per problem at a higher temperature (the MultiPL-E repository's own examples\nuse temperature 0.2 for pass@1 and temperature 0.8 with 200 samples for pass@10/pass@100). There is no\npartial credit: a completion either compiles/runs and passes every test, or it does not.\n\n## Dataset and licence\n\nThe dataset is published under the MIT licence. Item counts differ by language because a small number of\nthe original HumanEval and MBPP problems cannot be faithfully ported to every target: the HumanEval-\nderived sets run from 154 to 161 items (out of the original 164), and the MBPP-derived sets from 354 to\n397 items, per the nuprl/MultiPL-E dataset card. The original 2022 paper translated to 18 languages\nbeyond Python; the maintained repository has since grown via community contributions to cover roughly\ntwo dozen, adding languages such as Ada, Clojure, Dart, Elixir, Haskell and OCaml that were not in the\noriginal paper. All prompts and tests are public.\n\n## Who publishes it\n\nMultiPL-E comes from the Northeastern University Programming Research Lab (nuprl), led by Arjun Guha,\nwith co-authors from Wellesley College, Oberlin College and Stevens Institute of Technology: Federico\nCassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee,\nYangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, Michael Greenberg and Abhinav Jangda. The paper was\nposted to arXiv in August 2022; a revised version was later published in IEEE Transactions on Software\nEngineering. The nuprl GitHub organisation maintains the repository and dataset today.\n\n## Lineage\n\nMultiPL-E is a translation layer over HumanEval (OpenAI) and MBPP, not a new problem set of its own; it\ndoes not supersede either. Within this repository, its variants are the per-language subset pages:\n`multipl_e_cpp`, `multipl_e_csharp`, `multipl_e_go`, `multipl_e_java`, `multipl_e_javascript`,\n`multipl_e_julia`, `multipl_e_kotlin`, `multipl_e_lua`, `multipl_e_perl`, `multipl_e_php`,\n`multipl_e_python`, `multipl_e_r`, `multipl_e_ruby`, `multipl_e_rust`, `multipl_e_scala`,\n`multipl_e_swift` and `multipl_e_typescript`. The repository also carries translators for languages that\ndo not yet have a page here, including Ada, Bash, Clojure, D, Dart, Elixir, F#, Haskell, OCaml and\nRacket.\n\n## Saturation and contamination\n\nThere is no single ceiling for the family: MultiPL-E's own paper found that Codex matched or exceeded\nits Python pass rate on several other languages, and scores reported for current models spread widely\nbetween high-resource languages (Python, JavaScript, Java) and lower-resource ones (R, Perl, Lua), so a\nfamily-level average obscures more than it reveals. Contamination risk is high: HumanEval and MBPP have\nbeen public since 2021-2022 and are well represented in web-scale pretraining data, and because\nMultiPL-E keeps the same underlying problem logic across languages, memorising a Python solution gives a\nhead start on every translated version too.\n\n## How to run it\n\nThe reference implementation lives in the nuprl/MultiPL-E repository, which ships the per-language\ntranslators, prompt templates and Docker containers with each target language's toolchain. The\nbigcode-evaluation-harness project wraps the same dataset with task names of the form `multiple-<lang>`\n(for example `multiple-py`, `multiple-rs`, `multiple-cs`). Numbers are hard to compare across publishers\nwhen they differ on which languages are averaged together, the prompt format (some models are given a\ncompletion-style prompt, others a chat-style instruction), the stop tokens used to truncate generation,\nand the sampling budget used for pass@1 versus pass@10/pass@100.\n\n## Reading the numbers\n\nA single \"MultiPL-E score\" usually means an average over some subset of the supported languages, and\nthat subset is not standardised between leaderboards, so two vendors' MultiPL-E numbers are only\ncomparable if they picked the same languages. A high score on a high-resource language such as Python or\nJavaScript says less about generalisation than a comparable score on a lower-resource language such as R\nor Lua, where training data is scarcer and item counts are smaller (as few as 154), which also makes\nthose per-language scores noisier. Look at the per-language breakdown, not just the average, before\nconcluding a model \"generalises across languages.\"\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "multipl_e",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 86.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 85.7,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 84.9,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.6,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.6,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 84.6,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.6,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 83.0,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct",
   "model_id": "qwen/qwen2-5-coder-32b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.3,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 32B Instruct AWQ",
   "model_id": "qwen/qwen2-5-coder-32b-instruct-awq",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.3,
   "source": "provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 77.3,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.8,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Codestral (latest)",
   "model_id": "mistral/codestral-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 75.7,
   "source": "lmarena.ai, provider-reports"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 26B",
   "model_id": "google/gemma-4-26b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 75.4,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 14B Instruct",
   "model_id": "qwen/qwen2-5-coder-14b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74.1,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 72.9,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 72.9,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 72.0,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 72.0,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 70.8,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 70.8,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct",
   "model_id": "qwen/qwen2-5-coder-7b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 67.6,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 Coder 7B Instruct GPTQ Int4",
   "model_id": "qwen/qwen2-5-coder-7b-instruct-gptq-int4",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 67.6,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "CodeLlama 34B Instruct hf",
   "model_id": "meta/codellama-34b-instruct-hf",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 58.8,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard-v1"
  }
 ],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "The source problems (HumanEval, MBPP) have been public since 2021-2022 and are widely represented in web-scale pretraining corpora. Because MultiPL-E keeps the same underlying algorithmic problem and only changes the surface syntax, a model that has memorised a Python solution starts every translated version with an advantage; this is not specific to any one target language.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "C++",
    "C#",
    "Go",
    "Java",
    "JavaScript",
    "Julia",
    "Lua",
    "Perl",
    "PHP",
    "Python",
    "R",
    "Ruby",
    "Rust",
    "Scala",
    "Swift",
    "TypeScript"
   ],
   "license": "MIT",
   "modalities": [
    "code"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Sizes vary per target language because a handful of the original problems cannot be ported faithfully to every language. Reading the nuprl/MultiPL-E dataset card on 2026-09-07: the HumanEval-derived set ranges from 154 items (Go) to 161 items (C++, JavaScript, Lua, PHP, Perl, R, Ruby, Racket) out of the original 164; the MBPP-derived set ranges from 354 items (Rust) to 397 items (C++, JavaScript, Lua, PHP, R, Ruby, Racket) out of the original MBPP pool. There is no single \"MultiPL-E size\" that applies to every language.\n",
   "splits": "one 'test' split per language/source-benchmark pair (e.g. humaneval-rs, mbpp-rs); no train split",
   "url": "https://huggingface.co/datasets/nuprl/MultiPL-E"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice C",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "bigcode-evaluation-harness (task names multiple-<lang>, e.g. multiple-py, multiple-rs, multiple-cs); the nuprl/MultiPL-E repository provides the reference per-language translators and execution containers."
  },
  "id": "multipl_e",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "multipl_e_cpp",
    "multipl_e_csharp",
    "multipl_e_go",
    "multipl_e_java",
    "multipl_e_javascript",
    "multipl_e_julia",
    "multipl_e_kotlin",
    "multipl_e_lua",
    "multipl_e_perl",
    "multipl_e_php",
    "multipl_e_python",
    "multipl_e_r",
    "multipl_e_ruby",
    "multipl_e_rust",
    "multipl_e_scala",
    "multipl_e_swift",
    "multipl_e_typescript"
   ]
  },
  "measures": "MultiPL-E tests whether a code model's ability to solve short, self-contained function-completion problems in Python carries over to other programming languages. It takes the same underlying problems used by HumanEval (164 hand-written tasks) and MBPP (a larger set of crowd-written \"basic Python\" tasks) and mechanically translates each problem's signature, docstring and tests into a target language, leaving the algorithmic content unchanged. The model is given a function signature and docstring in the target language and must produce a working function body.\n",
  "metric": {
   "baseline_note": "No published human baseline; the source HumanEval and MBPP papers did not establish one, and MultiPL-E's translated items were not independently re-solved by humans.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "pass@1 (also pass@10, pass@100)",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "MultiPL-E",
  "page_kind": "family",
  "paper": {
   "arxiv": "2208.08227",
   "title": "MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation",
   "url": "https://arxiv.org/abs/2208.08227",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Federico Cassano",
    "John Gouwar",
    "Daniel Nguyen",
    "Sydney Nguyen",
    "Luna Phipps-Costin",
    "Donald Pinckney",
    "Ming-Ho Yee",
    "Yangtian Zi",
    "Carolyn Jane Anderson",
    "Molly Q Feldman",
    "Arjun Guha",
    "Michael Greenberg",
    "Abhinav Jangda"
   ],
   "org": "Northeastern University Programming Research Lab (nuprl), with co-authors from Wellesley College, Oberlin College and Stevens Institute of Technology",
   "url": "https://github.com/nuprl/MultiPL-E"
  },
  "released": "2022-08",
  "repo_url": "https://github.com/nuprl/MultiPL-E",
  "saturation": {
   "as_of": "",
   "note": "No single ceiling applies across the family: the original paper reported that Codex matched or exceeded its own Python pass rate on several other languages, and this repository's own per-language model-card scores (not used here as a source) show high-resource languages such as Python, JavaScript and Java sitting well above lower-resource ones such as R, Perl and Lua. A family-wide \"MultiPL-E score\" therefore hides a spread that a per-language score does not.\n",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "MultiPL-E: A Scalable and Extensible Approach to Benchmarking Neural Code Generation",
    "url": "https://arxiv.org/abs/2208.08227"
   },
   {
    "accessed": "2026-09-07",
    "title": "nuprl/MultiPL-E: A multi-programming language benchmark for LLMs",
    "url": "https://github.com/nuprl/MultiPL-E"
   },
   {
    "accessed": "2026-09-07",
    "title": "MultiPL-E docs: Introduction",
    "url": "https://github.com/nuprl/MultiPL-E/blob/main/docs/index.md"
   },
   {
    "accessed": "2026-09-07",
    "title": "nuprl/MultiPL-E dataset card",
    "url": "https://huggingface.co/datasets/nuprl/MultiPL-E"
   },
   {
    "accessed": "2026-09-07",
    "title": "bigcode-project/bigcode-evaluation-harness",
    "url": "https://github.com/bigcode-project/bigcode-evaluation-harness"
   }
  ],
  "status": "active",
  "subcategory": "multilingual code generation",
  "summary": "MultiPL-E mechanically translates the HumanEval and MBPP Python code-generation benchmarks into 18+ other programming languages and scores pass@k in each.",
  "tags": [
   "code-generation",
   "multilingual",
   "pass-at-k",
   "humaneval",
   "mbpp",
   "execution-based"
  ],
  "task_format": "Function completion: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body. The completion is inserted into a per-language test harness, compiled or interpreted with the language's real toolchain inside a container, and run against translated unit tests.\n"
 }
}