MultiPL-E: Ruby

The Ruby subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into Ruby and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into Ruby by rewriting each problem's function signature, docstring and tests with Ruby syntax and typing, then asks the model to complete the function body in Ruby. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in Ruby: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real Ruby toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI79.82026-04
GPT-5.1 ChatOpenAI79.82026-04
GPT-5.1 CodexOpenAI79.82026-04
GPT-5.1 Codex MaxOpenAI79.82026-04
GPT-5.1 Codex miniOpenAI79.82026-04
Claude Opus 4Anthropic78.52026-04
Claude Opus 4.6Anthropic78.52026-04
GPT-5OpenAI78.22026-04
GPT-5 Chat (latest)OpenAI78.22026-04
GPT-5 MiniOpenAI78.22026-04
GPT-5 NanoOpenAI78.22026-04
GPT-5 ProOpenAI78.22026-04
GPT-5-CodexOpenAI78.22026-04
GPT-5.2OpenAI78.22026-04
GPT-5.2 ChatOpenAI78.22026-04
GPT-5.2 CodexOpenAI78.22026-04
GPT-5.2 ProOpenAI78.22026-04
GPT-5.3 Chat (latest)OpenAI78.22026-04
GPT-5.3 CodexOpenAI78.22026-04
GPT-5.3 Codex SparkOpenAI78.22026-04
GPT-5.4OpenAI78.22026-04
GPT-5.4 miniOpenAI78.22026-04
GPT-5.4 nanoOpenAI78.22026-04
GPT-5.4 ProOpenAI78.22026-04
o3OpenAI77.52026-04
Claude Opus 4.5Anthropic77.22026-04
Claude Opus 4.5 (latest)Anthropic77.22026-04
Claude Sonnet 4.6Anthropic76.52026-04
Claude Opus 4.1Anthropic76.22026-04
Claude Opus 4.1 (latest)Anthropic76.22026-04
Claude Sonnet 4Anthropic75.22026-04
Claude Sonnet 4.5Anthropic75.22026-04
Claude Sonnet 4.5 (latest)Anthropic75.22026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team74.82026-04
Claude Opus 4 (latest)Anthropic74.52026-04
Gemini 2.5 ProGoogle DeepMind74.22026-04
GPT-4.1OpenAI73.52026-04
Grok 4xAI73.52026-04
Grok 4 FastxAI73.52026-04
Grok 4 Fast (Non-Reasoning)xAI73.52026-04
Grok 4.1 FastxAI73.52026-04
Grok 4.1 Fast (Non-Reasoning)xAI73.52026-04
Grok 4.20 (Non-Reasoning)xAI73.52026-04
Grok 4.20 (Reasoning)xAI73.52026-04
Grok 4.20 Multi-AgentxAI73.52026-04
Claude Sonnet 4 (latest)Anthropic72.52026-04
o4-miniOpenAI71.52026-04
o4-mini-deep-researchOpenAI71.52026-04
Claude Sonnet 3.7Anthropic70.52026-04
Qwen3 235B-A22BAlibaba / Qwen Team70.52026-04
GPT-4oOpenAI70.22026-04
GPT-4o (2024-05-13)OpenAI70.22026-04
GPT-4o (2024-08-06)OpenAI70.22026-04
GPT-4o (2024-11-20)OpenAI70.22026-04
DeepSeek V3.2 ExpDeepSeek69.52026-04
Claude Sonnet 3.5Anthropic68.52026-04
Claude Sonnet 3.5 v2Anthropic68.52026-04
DeepSeek V3.2DeepSeek68.52026-04
GPT-4 TurboOpenAI68.52026-04
o3-miniOpenAI68.52026-04
DeepSeek R1 0528DeepSeek67.22026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA67.22026-04
Claude Haiku 4.5Anthropic66.22026-04
Claude Haiku 4.5 (latest)Anthropic66.22026-04
DeepSeek R1DeepSeek65.52026-04
DeepSeek ReasonerDeepSeek65.52026-04
DeepSeek V3.1DeepSeek64.52026-04
Gemini 2.5 FlashGoogle DeepMind64.52026-04
Gemini 2.5 Flash ImageGoogle DeepMind64.52026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind64.52026-04
Gemini 2.5 Flash LiteGoogle DeepMind64.52026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind64.52026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind64.52026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind64.52026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind64.52026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind64.52026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind64.52026-04
Grok 3xAI64.52026-04
Grok 3 FastxAI64.52026-04
Grok 3 Fast LatestxAI64.52026-04
Grok 3 LatestxAI64.52026-04
Grok 3 MinixAI64.52026-04
Grok 3 Mini FastxAI64.52026-04
Grok 3 Mini Fast LatestxAI64.52026-04
Grok 3 Mini LatestxAI64.52026-04
Qwen3 32BAlibaba / Qwen Team63.52026-04
Qwen3 32B AWQAlibaba / Qwen Team63.52026-04
Qwen3 32B NVFP4NVIDIA63.52026-04
Claude Opus 3Anthropic62.52026-04
GPT-4OpenAI62.52026-04
GPT-4.1 miniOpenAI62.52026-04
GPT-4.1 nanoOpenAI62.52026-04
Mistral Large (latest)Mistral AI62.52026-04
Mistral Large 2.1Mistral AI62.52026-04
Mistral Large 3Mistral AI62.52026-04
DeepSeek V3 0324DeepSeek62.12026-04
DeepSeek ChatDeepSeek60.52026-04
DeepSeek V3DeepSeek60.52026-04
Llama 4 Maverick 17B 128E InstructMeta60.22026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta60.22026-04
GPT-4o miniOpenAI58.52026-04
Qwen2.5 72B InstructAlibaba / Qwen Team58.52026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team58.52026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team58.52026-04
Claude Haiku 3.5Anthropic58.22026-04
Claude Haiku 3.5 (latest)Anthropic58.22026-04
Gemma 4 31BGoogle DeepMind57.82026-04
gemma 4 31B itGoogle DeepMind57.82026-04
gemma 4 31B it GGUFUnsloth57.82026-04
Gemma 4 31B IT NVFP4NVIDIA57.82026-04
Llama 4 Scout 17B 16EMeta56.52026-04
Llama 4 Scout 17B 16E InstructMeta56.52026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta56.52026-04
Mistral Medium (latest)Mistral AI56.52026-04
Mistral Medium 3Mistral AI56.52026-04
Mistral Medium 3.1Mistral AI56.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA55.82026-04
Llama-3.3-70B-InstructMeta55.82026-04
Gemma 4 26BGoogle DeepMind55.52026-04
Codestral (latest)Mistral AI55.22026-04
phi 4Microsoft55.22026-04
Llama 3.1 70BMeta53.52026-04
Llama 3.1 70B InstructMeta53.52026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team50.22026-04
Mistral Small (latest)Mistral AI48.52026-04
Mistral Small 24B Instruct 2501Mistral AI48.52026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI48.52026-04
Mistral Small 3.2Mistral AI48.52026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI48.52026-04
Mistral Small 4Mistral AI48.52026-04
Mistral Small 4 119B 2603Mistral AI48.52026-04
Phi 4 mini instructMicrosoft42.52026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team42.52026-04
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team42.52026-04
CodeLlama 34B Instruct hfMeta38.52026-04

Data

This page as JSON · Edit on GitHub