MultiPL-E: TypeScript

The TypeScript subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into TypeScript and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into TypeScript by rewriting each problem's function signature, docstring and tests with TypeScript syntax and typing, then asks the model to complete the function body in TypeScript. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in TypeScript: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real TypeScript toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Claude Opus 4Anthropic90.82026-04
Claude Opus 4.6Anthropic90.82026-04
Claude Sonnet 4Anthropic88.22026-04
Claude Sonnet 4.5Anthropic88.22026-04
Claude Sonnet 4.5 (latest)Anthropic88.22026-04
GPT-4.1OpenAI87.12026-04
Gemini 2.5 ProGoogle DeepMind86.52026-04
DeepSeek R1DeepSeek86.12026-04
DeepSeek R1 0528DeepSeek86.12026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA86.12026-04
DeepSeek ReasonerDeepSeek86.12026-04
GPT-4oOpenAI84.22026-04
GPT-4o (2024-05-13)OpenAI84.22026-04
GPT-4o (2024-08-06)OpenAI84.22026-04
GPT-4o (2024-11-20)OpenAI84.22026-04
GPT-4o miniOpenAI84.22026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team81.22026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team81.22026-04
Gemma 4 31BGoogle DeepMind78.82026-04
gemma 4 31B itGoogle DeepMind78.82026-04
gemma 4 31B it GGUFUnsloth78.82026-04
Gemma 4 31B IT NVFP4NVIDIA78.82026-04
Mistral Large (latest)Mistral AI77.52026-04
Mistral Large 2.1Mistral AI77.52026-04
Mistral Large 3Mistral AI77.52026-04
Codestral (latest)Mistral AI77.12026-04
Gemma 4 26BGoogle DeepMind76.82026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team75.82026-04
Llama 3.3 70B Instruct NVFP4NVIDIA74.82026-04
Llama-3.3-70B-InstructMeta74.82026-04
phi 4Microsoft73.82026-04
Phi 4 mini instructMicrosoft73.82026-04
Llama 3.1 70BMeta72.52026-04
Llama 3.1 70B InstructMeta72.52026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team69.52026-04
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team69.52026-04
CodeLlama 34B Instruct hfMeta60.82026-04

Data

This page as JSON · Edit on GitHub