MultiPL-E: C#

The C# subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into C# and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into C# by rewriting each problem's function signature, docstring and tests with C# syntax and typing, then asks the model to complete the function body in C#. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in C#: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real C# toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI89.22026-04
GPT-5.1 ChatOpenAI89.22026-04
GPT-5.1 CodexOpenAI89.22026-04
GPT-5.1 Codex MaxOpenAI89.22026-04
GPT-5.1 Codex miniOpenAI89.22026-04
GPT-5OpenAI88.52026-04
GPT-5 Chat (latest)OpenAI88.52026-04
GPT-5 MiniOpenAI88.52026-04
GPT-5 NanoOpenAI88.52026-04
GPT-5 ProOpenAI88.52026-04
GPT-5-CodexOpenAI88.52026-04
GPT-5.2OpenAI88.52026-04
GPT-5.2 ChatOpenAI88.52026-04
GPT-5.2 CodexOpenAI88.52026-04
GPT-5.2 ProOpenAI88.52026-04
GPT-5.3 Chat (latest)OpenAI88.52026-04
GPT-5.3 CodexOpenAI88.52026-04
GPT-5.3 Codex SparkOpenAI88.52026-04
GPT-5.4OpenAI88.52026-04
GPT-5.4 miniOpenAI88.52026-04
GPT-5.4 nanoOpenAI88.52026-04
GPT-5.4 ProOpenAI88.52026-04
Claude Opus 4Anthropic88.22026-04
Claude Opus 4.6Anthropic88.22026-04
o3OpenAI87.82026-04
Claude Opus 4.5Anthropic87.52026-04
Claude Opus 4.5 (latest)Anthropic87.52026-04
Claude Sonnet 4.6Anthropic86.82026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team86.52026-04
Claude Opus 4.1Anthropic86.22026-04
Claude Opus 4.1 (latest)Anthropic86.22026-04
Claude Sonnet 4Anthropic85.52026-04
Claude Sonnet 4.5Anthropic85.52026-04
Claude Sonnet 4.5 (latest)Anthropic85.52026-04
GPT-4.1OpenAI85.22026-04
Grok 4xAI85.22026-04
Grok 4 FastxAI85.22026-04
Grok 4 Fast (Non-Reasoning)xAI85.22026-04
Grok 4.1 FastxAI85.22026-04
Grok 4.1 Fast (Non-Reasoning)xAI85.22026-04
Grok 4.20 (Non-Reasoning)xAI85.22026-04
Grok 4.20 (Reasoning)xAI85.22026-04
Grok 4.20 Multi-AgentxAI85.22026-04
Gemini 2.5 ProGoogle DeepMind84.82026-04
Claude Opus 4 (latest)Anthropic84.52026-04
o4-miniOpenAI84.52026-04
o4-mini-deep-researchOpenAI84.52026-04
Qwen3 235B-A22BAlibaba / Qwen Team83.52026-04
DeepSeek V3.2 ExpDeepSeek83.22026-04
Claude Sonnet 4 (latest)Anthropic82.52026-04
DeepSeek V3.2DeepSeek82.52026-04
o3-miniOpenAI82.52026-04
DeepSeek R1 0528DeepSeek82.12026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA82.12026-04
GPT-4oOpenAI82.12026-04
GPT-4o (2024-05-13)OpenAI82.12026-04
GPT-4o (2024-08-06)OpenAI82.12026-04
GPT-4o (2024-11-20)OpenAI82.12026-04
Claude Sonnet 3.7Anthropic81.52026-04
DeepSeek R1DeepSeek80.52026-04
DeepSeek ReasonerDeepSeek80.52026-04
GPT-4 TurboOpenAI80.52026-04
Claude Sonnet 3.5Anthropic80.22026-04
Claude Sonnet 3.5 v2Anthropic80.22026-04
DeepSeek V3.1DeepSeek79.82026-04
Claude Haiku 4.5Anthropic78.52026-04
Claude Haiku 4.5 (latest)Anthropic78.52026-04
Gemini 2.5 FlashGoogle DeepMind78.52026-04
Gemini 2.5 Flash ImageGoogle DeepMind78.52026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind78.52026-04
Gemini 2.5 Flash LiteGoogle DeepMind78.52026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind78.52026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind78.52026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind78.52026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind78.52026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind78.52026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind78.52026-04
Grok 3xAI78.52026-04
Grok 3 FastxAI78.52026-04
Grok 3 Fast LatestxAI78.52026-04
Grok 3 LatestxAI78.52026-04
Grok 3 MinixAI78.52026-04
Grok 3 Mini FastxAI78.52026-04
Grok 3 Mini Fast LatestxAI78.52026-04
Grok 3 Mini LatestxAI78.52026-04
Mistral Large (latest)Mistral AI78.22026-04
Mistral Large 2.1Mistral AI78.22026-04
Mistral Large 3Mistral AI78.22026-04
Qwen3 32BAlibaba / Qwen Team78.22026-04
Qwen3 32B AWQAlibaba / Qwen Team78.22026-04
Qwen3 32B NVFP4NVIDIA78.22026-04
DeepSeek V3 0324DeepSeek77.52026-04
GPT-4OpenAI76.52026-04
GPT-4.1 miniOpenAI76.52026-04
GPT-4.1 nanoOpenAI76.52026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team76.52026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team76.52026-04
DeepSeek ChatDeepSeek76.22026-04
DeepSeek V3DeepSeek76.22026-04
Claude Opus 3Anthropic75.52026-04
Llama 4 Maverick 17B 128E InstructMeta75.52026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta75.52026-04
Qwen2.5 72B InstructAlibaba / Qwen Team74.52026-04
Gemma 4 31BGoogle DeepMind74.22026-04
gemma 4 31B itGoogle DeepMind74.22026-04
gemma 4 31B it GGUFUnsloth74.22026-04
Gemma 4 31B IT NVFP4NVIDIA74.22026-04
Codestral (latest)Mistral AI73.52026-04
Llama 4 Scout 17B 16EMeta72.82026-04
Llama 4 Scout 17B 16E InstructMeta72.82026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta72.82026-04
phi 4Microsoft72.82026-04
Claude Haiku 3.5Anthropic72.52026-04
Claude Haiku 3.5 (latest)Anthropic72.52026-04
Gemma 4 26BGoogle DeepMind72.52026-04
GPT-4o miniOpenAI72.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA72.52026-04
Llama-3.3-70B-InstructMeta72.52026-04
Mistral Medium (latest)Mistral AI72.52026-04
Mistral Medium 3Mistral AI72.52026-04
Mistral Medium 3.1Mistral AI72.52026-04
Llama 3.1 70BMeta70.22026-04
Llama 3.1 70B InstructMeta70.22026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team68.22026-04
Mistral Small (latest)Mistral AI65.22026-04
Mistral Small 24B Instruct 2501Mistral AI65.22026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI65.22026-04
Mistral Small 3.2Mistral AI65.22026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI65.22026-04
Mistral Small 4Mistral AI65.22026-04
Mistral Small 4 119B 2603Mistral AI65.22026-04
Phi 4 mini instructMicrosoft62.52026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team60.82026-04
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team60.82026-04
CodeLlama 34B Instruct hfMeta55.22026-04

Data

This page as JSON · Edit on GitHub