MultiPL-E: R

The R subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into R and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into R by rewriting each problem's function signature, docstring and tests with R syntax and typing, then asks the model to complete the function body in R. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in R: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real R toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI69.52026-04
GPT-5.1 ChatOpenAI69.52026-04
GPT-5.1 CodexOpenAI69.52026-04
GPT-5.1 Codex MaxOpenAI69.52026-04
GPT-5.1 Codex miniOpenAI69.52026-04
Claude Opus 4Anthropic68.52026-04
Claude Opus 4.6Anthropic68.52026-04
GPT-5OpenAI68.22026-04
GPT-5 Chat (latest)OpenAI68.22026-04
GPT-5 MiniOpenAI68.22026-04
GPT-5 NanoOpenAI68.22026-04
GPT-5 ProOpenAI68.22026-04
GPT-5-CodexOpenAI68.22026-04
GPT-5.2OpenAI68.22026-04
GPT-5.2 ChatOpenAI68.22026-04
GPT-5.2 CodexOpenAI68.22026-04
GPT-5.2 ProOpenAI68.22026-04
GPT-5.3 Chat (latest)OpenAI68.22026-04
GPT-5.3 CodexOpenAI68.22026-04
GPT-5.3 Codex SparkOpenAI68.22026-04
GPT-5.4OpenAI68.22026-04
GPT-5.4 miniOpenAI68.22026-04
GPT-5.4 nanoOpenAI68.22026-04
GPT-5.4 ProOpenAI68.22026-04
Claude Opus 4.5Anthropic67.82026-04
Claude Opus 4.5 (latest)Anthropic67.82026-04
o3OpenAI67.52026-04
Claude Sonnet 4.6Anthropic66.82026-04
Claude Opus 4.1Anthropic66.52026-04
Claude Opus 4.1 (latest)Anthropic66.52026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team66.52026-04
Claude Sonnet 4Anthropic65.22026-04
Claude Sonnet 4.5Anthropic65.22026-04
Claude Sonnet 4.5 (latest)Anthropic65.22026-04
Gemini 2.5 ProGoogle DeepMind64.82026-04
Grok 4xAI64.52026-04
Grok 4 FastxAI64.52026-04
Grok 4 Fast (Non-Reasoning)xAI64.52026-04
Grok 4.1 FastxAI64.52026-04
Grok 4.1 Fast (Non-Reasoning)xAI64.52026-04
Grok 4.20 (Non-Reasoning)xAI64.52026-04
Grok 4.20 (Reasoning)xAI64.52026-04
Grok 4.20 Multi-AgentxAI64.52026-04
Claude Opus 4 (latest)Anthropic64.22026-04
o4-miniOpenAI63.82026-04
o4-mini-deep-researchOpenAI63.82026-04
GPT-4.1OpenAI63.52026-04
Qwen3 235B-A22BAlibaba / Qwen Team62.52026-04
Claude Sonnet 4 (latest)Anthropic62.12026-04
DeepSeek V3.2 ExpDeepSeek61.52026-04
Claude Sonnet 3.7Anthropic60.52026-04
GPT-4oOpenAI60.52026-04
GPT-4o (2024-05-13)OpenAI60.52026-04
GPT-4o (2024-08-06)OpenAI60.52026-04
GPT-4o (2024-11-20)OpenAI60.52026-04
o3-miniOpenAI60.52026-04
DeepSeek R1 0528DeepSeek60.22026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA60.22026-04
DeepSeek V3.2DeepSeek60.22026-04
DeepSeek R1DeepSeek58.52026-04
DeepSeek ReasonerDeepSeek58.52026-04
Claude Sonnet 3.5Anthropic58.22026-04
Claude Sonnet 3.5 v2Anthropic58.22026-04
GPT-4 TurboOpenAI58.22026-04
Claude Haiku 4.5Anthropic56.82026-04
Claude Haiku 4.5 (latest)Anthropic56.82026-04
Grok 3xAI56.82026-04
Grok 3 FastxAI56.82026-04
Grok 3 Fast LatestxAI56.82026-04
Grok 3 LatestxAI56.82026-04
Grok 3 MinixAI56.82026-04
Grok 3 Mini FastxAI56.82026-04
Grok 3 Mini Fast LatestxAI56.82026-04
Grok 3 Mini LatestxAI56.82026-04
DeepSeek V3.1DeepSeek56.52026-04
Gemini 2.5 FlashGoogle DeepMind56.22026-04
Gemini 2.5 Flash ImageGoogle DeepMind56.22026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind56.22026-04
Gemini 2.5 Flash LiteGoogle DeepMind56.22026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind56.22026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind56.22026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind56.22026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind56.22026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind56.22026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind56.22026-04
Qwen3 32BAlibaba / Qwen Team55.52026-04
Qwen3 32B AWQAlibaba / Qwen Team55.52026-04
Qwen3 32B NVFP4NVIDIA55.52026-04
Mistral Large (latest)Mistral AI55.22026-04
Mistral Large 2.1Mistral AI55.22026-04
Mistral Large 3Mistral AI55.22026-04
DeepSeek V3 0324DeepSeek54.22026-04
GPT-4OpenAI52.82026-04
GPT-4.1 miniOpenAI52.82026-04
GPT-4.1 nanoOpenAI52.82026-04
Claude Opus 3Anthropic52.52026-04
DeepSeek ChatDeepSeek52.52026-04
DeepSeek V3DeepSeek52.52026-04
Llama 4 Maverick 17B 128E InstructMeta52.22026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta52.22026-04
Gemma 4 31BGoogle DeepMind50.52026-04
gemma 4 31B itGoogle DeepMind50.52026-04
gemma 4 31B it GGUFUnsloth50.52026-04
Gemma 4 31B IT NVFP4NVIDIA50.52026-04
Qwen2.5 72B InstructAlibaba / Qwen Team50.52026-04
GPT-4o miniOpenAI50.22026-04
Llama 4 Scout 17B 16EMeta48.82026-04
Llama 4 Scout 17B 16E InstructMeta48.82026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta48.82026-04
Claude Haiku 3.5Anthropic48.52026-04
Claude Haiku 3.5 (latest)Anthropic48.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA48.52026-04
Llama-3.3-70B-InstructMeta48.52026-04
Mistral Medium (latest)Mistral AI48.52026-04
Mistral Medium 3Mistral AI48.52026-04
Mistral Medium 3.1Mistral AI48.52026-04
Gemma 4 26BGoogle DeepMind48.22026-04
phi 4Microsoft48.22026-04
Llama 3.1 70BMeta46.52026-04
Llama 3.1 70B InstructMeta46.52026-04
Mistral Small (latest)Mistral AI40.52026-04
Mistral Small 24B Instruct 2501Mistral AI40.52026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI40.52026-04
Mistral Small 3.2Mistral AI40.52026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI40.52026-04
Mistral Small 4Mistral AI40.52026-04
Mistral Small 4 119B 2603Mistral AI40.52026-04
Phi 4 mini instructMicrosoft36.52026-04
CodeLlama 34B Instruct hfMeta28.52026-04

Data

This page as JSON · Edit on GitHub