MultiPL-E: Scala

The Scala subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into Scala and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into Scala by rewriting each problem's function signature, docstring and tests with Scala syntax and typing, then asks the model to complete the function body in Scala. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in Scala: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real Scala toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI73.82026-04
GPT-5.1 ChatOpenAI73.82026-04
GPT-5.1 CodexOpenAI73.82026-04
GPT-5.1 Codex MaxOpenAI73.82026-04
GPT-5.1 Codex miniOpenAI73.82026-04
GPT-5OpenAI72.52026-04
GPT-5 Chat (latest)OpenAI72.52026-04
GPT-5 MiniOpenAI72.52026-04
GPT-5 NanoOpenAI72.52026-04
GPT-5 ProOpenAI72.52026-04
GPT-5-CodexOpenAI72.52026-04
GPT-5.2OpenAI72.52026-04
GPT-5.2 ChatOpenAI72.52026-04
GPT-5.2 CodexOpenAI72.52026-04
GPT-5.2 ProOpenAI72.52026-04
GPT-5.3 Chat (latest)OpenAI72.52026-04
GPT-5.3 CodexOpenAI72.52026-04
GPT-5.3 Codex SparkOpenAI72.52026-04
GPT-5.4OpenAI72.52026-04
GPT-5.4 miniOpenAI72.52026-04
GPT-5.4 nanoOpenAI72.52026-04
GPT-5.4 ProOpenAI72.52026-04
Claude Opus 4Anthropic72.22026-04
Claude Opus 4.6Anthropic72.22026-04
o3OpenAI71.82026-04
Claude Opus 4.5Anthropic71.52026-04
Claude Opus 4.5 (latest)Anthropic71.52026-04
Claude Sonnet 4.6Anthropic70.22026-04
Claude Opus 4.1Anthropic70.12026-04
Claude Opus 4.1 (latest)Anthropic70.12026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team69.52026-04
Claude Sonnet 4Anthropic68.52026-04
Claude Sonnet 4.5Anthropic68.52026-04
Claude Sonnet 4.5 (latest)Anthropic68.52026-04
Gemini 2.5 ProGoogle DeepMind68.52026-04
Grok 4xAI68.22026-04
Grok 4 FastxAI68.22026-04
Grok 4 Fast (Non-Reasoning)xAI68.22026-04
Grok 4.1 FastxAI68.22026-04
Grok 4.1 Fast (Non-Reasoning)xAI68.22026-04
Grok 4.20 (Non-Reasoning)xAI68.22026-04
Grok 4.20 (Reasoning)xAI68.22026-04
Grok 4.20 Multi-AgentxAI68.22026-04
Claude Opus 4 (latest)Anthropic67.82026-04
GPT-4.1OpenAI67.22026-04
o4-miniOpenAI66.22026-04
o4-mini-deep-researchOpenAI66.22026-04
Claude Sonnet 4 (latest)Anthropic65.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team65.22026-04
DeepSeek V3.2 ExpDeepSeek64.52026-04
GPT-4oOpenAI64.52026-04
GPT-4o (2024-05-13)OpenAI64.52026-04
GPT-4o (2024-08-06)OpenAI64.52026-04
GPT-4o (2024-11-20)OpenAI64.52026-04
Claude Sonnet 3.7Anthropic64.22026-04
DeepSeek V3.2DeepSeek63.22026-04
o3-miniOpenAI63.22026-04
GPT-4 TurboOpenAI62.82026-04
Claude Sonnet 3.5Anthropic62.52026-04
Claude Sonnet 3.5 v2Anthropic62.52026-04
DeepSeek R1 0528DeepSeek62.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA62.52026-04
Claude Haiku 4.5Anthropic60.52026-04
Claude Haiku 4.5 (latest)Anthropic60.52026-04
DeepSeek R1DeepSeek60.22026-04
DeepSeek ReasonerDeepSeek60.22026-04
DeepSeek V3.1DeepSeek59.82026-04
Gemini 2.5 FlashGoogle DeepMind58.82026-04
Gemini 2.5 Flash ImageGoogle DeepMind58.82026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind58.82026-04
Gemini 2.5 Flash LiteGoogle DeepMind58.82026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind58.82026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind58.82026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind58.82026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind58.82026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind58.82026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind58.82026-04
Grok 3xAI58.82026-04
Grok 3 FastxAI58.82026-04
Grok 3 Fast LatestxAI58.82026-04
Grok 3 LatestxAI58.82026-04
Grok 3 MinixAI58.82026-04
Grok 3 Mini FastxAI58.82026-04
Grok 3 Mini Fast LatestxAI58.82026-04
Grok 3 Mini LatestxAI58.82026-04
Qwen3 32BAlibaba / Qwen Team58.22026-04
Qwen3 32B AWQAlibaba / Qwen Team58.22026-04
Qwen3 32B NVFP4NVIDIA58.22026-04
Mistral Large (latest)Mistral AI57.82026-04
Mistral Large 2.1Mistral AI57.82026-04
Mistral Large 3Mistral AI57.82026-04
DeepSeek V3 0324DeepSeek57.22026-04
GPT-4OpenAI56.82026-04
GPT-4.1 miniOpenAI56.82026-04
GPT-4.1 nanoOpenAI56.82026-04
Claude Opus 3Anthropic56.22026-04
DeepSeek ChatDeepSeek55.82026-04
DeepSeek V3DeepSeek55.82026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team55.22026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team55.22026-04
Llama 4 Maverick 17B 128E InstructMeta54.52026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta54.52026-04
Qwen2.5 72B InstructAlibaba / Qwen Team53.22026-04
GPT-4o miniOpenAI52.82026-04
Claude Haiku 3.5Anthropic52.52026-04
Claude Haiku 3.5 (latest)Anthropic52.52026-04
Gemma 4 31BGoogle DeepMind52.52026-04
gemma 4 31B itGoogle DeepMind52.52026-04
gemma 4 31B it GGUFUnsloth52.52026-04
Gemma 4 31B IT NVFP4NVIDIA52.52026-04
Codestral (latest)Mistral AI52.12026-04
Mistral Medium (latest)Mistral AI51.22026-04
Mistral Medium 3Mistral AI51.22026-04
Mistral Medium 3.1Mistral AI51.22026-04
Llama 4 Scout 17B 16EMeta50.82026-04
Llama 4 Scout 17B 16E InstructMeta50.82026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta50.82026-04
phi 4Microsoft50.52026-04
Gemma 4 26BGoogle DeepMind50.22026-04
Llama 3.3 70B Instruct NVFP4NVIDIA50.22026-04
Llama-3.3-70B-InstructMeta50.22026-04
Llama 3.1 70BMeta48.22026-04
Llama 3.1 70B InstructMeta48.22026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team47.52026-04
Mistral Small (latest)Mistral AI43.22026-04
Mistral Small 24B Instruct 2501Mistral AI43.22026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI43.22026-04
Mistral Small 3.2Mistral AI43.22026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI43.22026-04
Mistral Small 4Mistral AI43.22026-04
Mistral Small 4 119B 2603Mistral AI43.22026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team40.22026-04
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team40.22026-04
Phi 4 mini instructMicrosoft38.22026-04
CodeLlama 34B Instruct hfMeta32.52026-04

Data

This page as JSON · Edit on GitHub