MultiPL-E: PHP

The PHP subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into PHP and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into PHP by rewriting each problem's function signature, docstring and tests with PHP syntax and typing, then asks the model to complete the function body in PHP. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in PHP: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real PHP toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI86.52026-04
GPT-5.1 ChatOpenAI86.52026-04
GPT-5.1 CodexOpenAI86.52026-04
GPT-5.1 Codex MaxOpenAI86.52026-04
GPT-5.1 Codex miniOpenAI86.52026-04
Claude Opus 4Anthropic85.52026-04
Claude Opus 4.6Anthropic85.52026-04
GPT-5OpenAI85.22026-04
GPT-5 Chat (latest)OpenAI85.22026-04
GPT-5 MiniOpenAI85.22026-04
GPT-5 NanoOpenAI85.22026-04
GPT-5 ProOpenAI85.22026-04
GPT-5-CodexOpenAI85.22026-04
GPT-5.2OpenAI85.22026-04
GPT-5.2 ChatOpenAI85.22026-04
GPT-5.2 CodexOpenAI85.22026-04
GPT-5.2 ProOpenAI85.22026-04
GPT-5.3 Chat (latest)OpenAI85.22026-04
GPT-5.3 CodexOpenAI85.22026-04
GPT-5.3 Codex SparkOpenAI85.22026-04
GPT-5.4OpenAI85.22026-04
GPT-5.4 miniOpenAI85.22026-04
GPT-5.4 nanoOpenAI85.22026-04
GPT-5.4 ProOpenAI85.22026-04
o3OpenAI85.12026-04
Claude Opus 4.5Anthropic84.82026-04
Claude Opus 4.5 (latest)Anthropic84.82026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team84.22026-04
Claude Opus 4.1Anthropic83.52026-04
Claude Opus 4.1 (latest)Anthropic83.52026-04
Claude Sonnet 4.6Anthropic83.52026-04
GPT-4.1OpenAI82.82026-04
Grok 4xAI82.82026-04
Grok 4 FastxAI82.82026-04
Grok 4 Fast (Non-Reasoning)xAI82.82026-04
Grok 4.1 FastxAI82.82026-04
Grok 4.1 Fast (Non-Reasoning)xAI82.82026-04
Grok 4.20 (Non-Reasoning)xAI82.82026-04
Grok 4.20 (Reasoning)xAI82.82026-04
Grok 4.20 Multi-AgentxAI82.82026-04
Gemini 2.5 ProGoogle DeepMind82.52026-04
Claude Sonnet 4Anthropic82.22026-04
Claude Sonnet 4.5Anthropic82.22026-04
Claude Sonnet 4.5 (latest)Anthropic82.22026-04
o4-miniOpenAI82.12026-04
o4-mini-deep-researchOpenAI82.12026-04
Claude Opus 4 (latest)Anthropic81.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team81.22026-04
DeepSeek V3.2 ExpDeepSeek80.82026-04
GPT-4oOpenAI80.52026-04
GPT-4o (2024-05-13)OpenAI80.52026-04
GPT-4o (2024-08-06)OpenAI80.52026-04
GPT-4o (2024-11-20)OpenAI80.52026-04
DeepSeek V3.2DeepSeek80.22026-04
o3-miniOpenAI80.22026-04
Claude Sonnet 4 (latest)Anthropic79.82026-04
DeepSeek R1 0528DeepSeek79.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA79.52026-04
Claude Sonnet 3.7Anthropic78.82026-04
GPT-4 TurboOpenAI78.22026-04
DeepSeek R1DeepSeek77.82026-04
DeepSeek ReasonerDeepSeek77.82026-04
Claude Sonnet 3.5Anthropic77.52026-04
Claude Sonnet 3.5 v2Anthropic77.52026-04
DeepSeek V3.1DeepSeek77.22026-04
Gemini 2.5 FlashGoogle DeepMind76.22026-04
Gemini 2.5 Flash ImageGoogle DeepMind76.22026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind76.22026-04
Gemini 2.5 Flash LiteGoogle DeepMind76.22026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind76.22026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind76.22026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind76.22026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind76.22026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind76.22026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind76.22026-04
Grok 3xAI76.22026-04
Grok 3 FastxAI76.22026-04
Grok 3 Fast LatestxAI76.22026-04
Grok 3 LatestxAI76.22026-04
Grok 3 MinixAI76.22026-04
Grok 3 Mini FastxAI76.22026-04
Grok 3 Mini Fast LatestxAI76.22026-04
Grok 3 Mini LatestxAI76.22026-04
Claude Haiku 4.5Anthropic75.82026-04
Claude Haiku 4.5 (latest)Anthropic75.82026-04
Qwen3 32BAlibaba / Qwen Team75.82026-04
Qwen3 32B AWQAlibaba / Qwen Team75.82026-04
Qwen3 32B NVFP4NVIDIA75.82026-04
Mistral Large (latest)Mistral AI75.52026-04
Mistral Large 2.1Mistral AI75.52026-04
Mistral Large 3Mistral AI75.52026-04
DeepSeek V3 0324DeepSeek74.82026-04
GPT-4OpenAI74.22026-04
GPT-4.1 miniOpenAI74.22026-04
GPT-4.1 nanoOpenAI74.22026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team74.22026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team74.22026-04
DeepSeek ChatDeepSeek73.52026-04
DeepSeek V3DeepSeek73.52026-04
Llama 4 Maverick 17B 128E InstructMeta73.22026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta73.22026-04
Claude Opus 3Anthropic72.82026-04
Qwen2.5 72B InstructAlibaba / Qwen Team72.22026-04
Gemma 4 31BGoogle DeepMind71.52026-04
gemma 4 31B itGoogle DeepMind71.52026-04
gemma 4 31B it GGUFUnsloth71.52026-04
Gemma 4 31B IT NVFP4NVIDIA71.52026-04
Codestral (latest)Mistral AI71.22026-04
Llama 4 Scout 17B 16EMeta70.52026-04
Llama 4 Scout 17B 16E InstructMeta70.52026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta70.52026-04
GPT-4o miniOpenAI70.22026-04
Llama 3.3 70B Instruct NVFP4NVIDIA70.22026-04
Llama-3.3-70B-InstructMeta70.22026-04
Mistral Medium (latest)Mistral AI70.22026-04
Mistral Medium 3Mistral AI70.22026-04
Mistral Medium 3.1Mistral AI70.22026-04
Gemma 4 26BGoogle DeepMind69.82026-04
phi 4Microsoft69.52026-04
Claude Haiku 3.5Anthropic68.82026-04
Claude Haiku 3.5 (latest)Anthropic68.82026-04
Llama 3.1 70BMeta68.52026-04
Llama 3.1 70B InstructMeta68.52026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team66.52026-04
Mistral Small (latest)Mistral AI62.52026-04
Mistral Small 24B Instruct 2501Mistral AI62.52026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI62.52026-04
Mistral Small 3.2Mistral AI62.52026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI62.52026-04
Mistral Small 4Mistral AI62.52026-04
Mistral Small 4 119B 2603Mistral AI62.52026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team59.02026-04
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team59.02026-04
Phi 4 mini instructMicrosoft58.82026-04
CodeLlama 34B Instruct hfMeta52.82026-04

Data

This page as JSON · Edit on GitHub