MultiPL-E: Julia

The Julia subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into Julia and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into Julia by rewriting each problem's function signature, docstring and tests with Julia syntax and typing, then asks the model to complete the function body in Julia. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in Julia: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real Julia toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI71.82026-04
GPT-5.1 ChatOpenAI71.82026-04
GPT-5.1 CodexOpenAI71.82026-04
GPT-5.1 Codex MaxOpenAI71.82026-04
GPT-5.1 Codex miniOpenAI71.82026-04
GPT-5OpenAI70.52026-04
GPT-5 Chat (latest)OpenAI70.52026-04
GPT-5 MiniOpenAI70.52026-04
GPT-5 NanoOpenAI70.52026-04
GPT-5 ProOpenAI70.52026-04
GPT-5-CodexOpenAI70.52026-04
GPT-5.2OpenAI70.52026-04
GPT-5.2 ChatOpenAI70.52026-04
GPT-5.2 CodexOpenAI70.52026-04
GPT-5.2 ProOpenAI70.52026-04
GPT-5.3 Chat (latest)OpenAI70.52026-04
GPT-5.3 CodexOpenAI70.52026-04
GPT-5.3 Codex SparkOpenAI70.52026-04
GPT-5.4OpenAI70.52026-04
GPT-5.4 miniOpenAI70.52026-04
GPT-5.4 nanoOpenAI70.52026-04
GPT-5.4 ProOpenAI70.52026-04
Claude Opus 4Anthropic70.22026-04
Claude Opus 4.6Anthropic70.22026-04
o3OpenAI69.82026-04
Claude Opus 4.5Anthropic69.52026-04
Claude Opus 4.5 (latest)Anthropic69.52026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team69.22026-04
Claude Sonnet 4.6Anthropic68.52026-04
Claude Opus 4.1Anthropic68.22026-04
Claude Opus 4.1 (latest)Anthropic68.22026-04
Grok 4xAI67.82026-04
Grok 4 FastxAI67.82026-04
Grok 4 Fast (Non-Reasoning)xAI67.82026-04
Grok 4.1 FastxAI67.82026-04
Grok 4.1 Fast (Non-Reasoning)xAI67.82026-04
Grok 4.20 (Non-Reasoning)xAI67.82026-04
Grok 4.20 (Reasoning)xAI67.82026-04
Grok 4.20 Multi-AgentxAI67.82026-04
Claude Sonnet 4Anthropic67.52026-04
Claude Sonnet 4.5Anthropic67.52026-04
Claude Sonnet 4.5 (latest)Anthropic67.52026-04
Gemini 2.5 ProGoogle DeepMind67.22026-04
o4-miniOpenAI67.22026-04
o4-mini-deep-researchOpenAI67.22026-04
Claude Opus 4 (latest)Anthropic66.52026-04
GPT-4.1OpenAI65.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team65.82026-04
DeepSeek V3.2 ExpDeepSeek65.52026-04
Claude Sonnet 4 (latest)Anthropic64.52026-04
DeepSeek R1 0528DeepSeek64.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA64.52026-04
DeepSeek V3.2DeepSeek64.22026-04
o3-miniOpenAI64.22026-04
Claude Sonnet 3.7Anthropic63.22026-04
GPT-4oOpenAI62.82026-04
GPT-4o (2024-05-13)OpenAI62.82026-04
GPT-4o (2024-08-06)OpenAI62.82026-04
GPT-4o (2024-11-20)OpenAI62.82026-04
DeepSeek R1DeepSeek62.12026-04
DeepSeek ReasonerDeepSeek62.12026-04
GPT-4 TurboOpenAI61.82026-04
Claude Sonnet 3.5Anthropic61.52026-04
Claude Sonnet 3.5 v2Anthropic61.52026-04
Gemini 2.5 FlashGoogle DeepMind60.52026-04
Gemini 2.5 Flash ImageGoogle DeepMind60.52026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind60.52026-04
Gemini 2.5 Flash LiteGoogle DeepMind60.52026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind60.52026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind60.52026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind60.52026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind60.52026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind60.52026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind60.52026-04
Claude Haiku 4.5Anthropic60.22026-04
Claude Haiku 4.5 (latest)Anthropic60.22026-04
DeepSeek V3.1DeepSeek60.22026-04
Grok 3xAI60.22026-04
Grok 3 FastxAI60.22026-04
Grok 3 Fast LatestxAI60.22026-04
Grok 3 LatestxAI60.22026-04
Grok 3 MinixAI60.22026-04
Grok 3 Mini FastxAI60.22026-04
Grok 3 Mini Fast LatestxAI60.22026-04
Grok 3 Mini LatestxAI60.22026-04
Qwen3 32BAlibaba / Qwen Team59.22026-04
Qwen3 32B AWQAlibaba / Qwen Team59.22026-04
Qwen3 32B NVFP4NVIDIA59.22026-04
Mistral Large (latest)Mistral AI58.82026-04
Mistral Large 2.1Mistral AI58.82026-04
Mistral Large 3Mistral AI58.82026-04
DeepSeek V3 0324DeepSeek58.52026-04
DeepSeek ChatDeepSeek56.82026-04
DeepSeek V3DeepSeek56.82026-04
GPT-4OpenAI56.22026-04
GPT-4.1 miniOpenAI56.22026-04
GPT-4.1 nanoOpenAI56.22026-04
Claude Opus 3Anthropic55.82026-04
Llama 4 Maverick 17B 128E InstructMeta55.82026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta55.82026-04
Qwen2.5 72B InstructAlibaba / Qwen Team54.82026-04
Gemma 4 31BGoogle DeepMind54.22026-04
gemma 4 31B itGoogle DeepMind54.22026-04
gemma 4 31B it GGUFUnsloth54.22026-04
Gemma 4 31B IT NVFP4NVIDIA54.22026-04
GPT-4o miniOpenAI53.52026-04
Gemma 4 26BGoogle DeepMind52.52026-04
Llama 4 Scout 17B 16EMeta52.52026-04
Llama 4 Scout 17B 16E InstructMeta52.52026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta52.52026-04
Mistral Medium (latest)Mistral AI52.52026-04
Mistral Medium 3Mistral AI52.52026-04
Mistral Medium 3.1Mistral AI52.52026-04
phi 4Microsoft52.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA52.22026-04
Llama-3.3-70B-InstructMeta52.22026-04
Claude Haiku 3.5Anthropic51.82026-04
Claude Haiku 3.5 (latest)Anthropic51.82026-04
Llama 3.1 70BMeta50.52026-04
Llama 3.1 70B InstructMeta50.52026-04
Mistral Small (latest)Mistral AI44.22026-04
Mistral Small 24B Instruct 2501Mistral AI44.22026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI44.22026-04
Mistral Small 3.2Mistral AI44.22026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI44.22026-04
Mistral Small 4Mistral AI44.22026-04
Mistral Small 4 119B 2603Mistral AI44.22026-04
Phi 4 mini instructMicrosoft40.22026-04
CodeLlama 34B Instruct hfMeta32.82026-04

Data

This page as JSON · Edit on GitHub