MultiPL-E: Lua

The Lua subset of MultiPL-E: HumanEval and MBPP function-completion problems translated into Lua and scored with pass@1.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategorymultilingual code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset licenceMIT
PublisherNortheastern University Programming Research Lab (nuprl)

What it measures

This subset translates the HumanEval and MBPP prompts into Lua by rewriting each problem's function signature, docstring and tests with Lua syntax and typing, then asks the model to complete the function body in Lua. The underlying algorithmic problem is unchanged from the Python original; only the surface language differs.

Task format

Function completion in Lua: given a translated signature, docstring and (for HumanEval-derived items) doctests, the model generates a function body, which is compiled or interpreted with a real Lua toolchain inside a container and checked against translated unit tests.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI66.82026-04
GPT-5.1 ChatOpenAI66.82026-04
GPT-5.1 CodexOpenAI66.82026-04
GPT-5.1 Codex MaxOpenAI66.82026-04
GPT-5.1 Codex miniOpenAI66.82026-04
Claude Opus 4Anthropic65.82026-04
Claude Opus 4.6Anthropic65.82026-04
GPT-5OpenAI65.52026-04
GPT-5 Chat (latest)OpenAI65.52026-04
GPT-5 MiniOpenAI65.52026-04
GPT-5 NanoOpenAI65.52026-04
GPT-5 ProOpenAI65.52026-04
GPT-5-CodexOpenAI65.52026-04
GPT-5.2OpenAI65.52026-04
GPT-5.2 ChatOpenAI65.52026-04
GPT-5.2 CodexOpenAI65.52026-04
GPT-5.2 ProOpenAI65.52026-04
GPT-5.3 Chat (latest)OpenAI65.52026-04
GPT-5.3 CodexOpenAI65.52026-04
GPT-5.3 Codex SparkOpenAI65.52026-04
GPT-5.4OpenAI65.52026-04
GPT-5.4 miniOpenAI65.52026-04
GPT-5.4 nanoOpenAI65.52026-04
GPT-5.4 ProOpenAI65.52026-04
o3OpenAI65.22026-04
Claude Opus 4.5Anthropic64.52026-04
Claude Opus 4.5 (latest)Anthropic64.52026-04
Claude Sonnet 4.6Anthropic63.82026-04
Claude Opus 4.1Anthropic63.52026-04
Claude Opus 4.1 (latest)Anthropic63.52026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team62.52026-04
Claude Sonnet 4Anthropic62.12026-04
Claude Sonnet 4.5Anthropic62.12026-04
Claude Sonnet 4.5 (latest)Anthropic62.12026-04
Gemini 2.5 ProGoogle DeepMind61.82026-04
GPT-4.1OpenAI61.52026-04
o4-miniOpenAI61.52026-04
o4-mini-deep-researchOpenAI61.52026-04
Claude Opus 4 (latest)Anthropic61.22026-04
Grok 4xAI61.22026-04
Grok 4 FastxAI61.22026-04
Grok 4 Fast (Non-Reasoning)xAI61.22026-04
Grok 4.1 FastxAI61.22026-04
Grok 4.1 Fast (Non-Reasoning)xAI61.22026-04
Grok 4.20 (Non-Reasoning)xAI61.22026-04
Grok 4.20 (Reasoning)xAI61.22026-04
Grok 4.20 Multi-AgentxAI61.22026-04
DeepSeek V3.2 ExpDeepSeek59.22026-04
Claude Sonnet 4 (latest)Anthropic58.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team58.82026-04
GPT-4oOpenAI58.52026-04
GPT-4o (2024-05-13)OpenAI58.52026-04
GPT-4o (2024-08-06)OpenAI58.52026-04
GPT-4o (2024-11-20)OpenAI58.52026-04
o3-miniOpenAI58.52026-04
DeepSeek V3.2DeepSeek58.22026-04
Claude Sonnet 3.7Anthropic57.82026-04
DeepSeek R1 0528DeepSeek57.82026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA57.82026-04
GPT-4 TurboOpenAI56.82026-04
Claude Sonnet 3.5Anthropic55.82026-04
Claude Sonnet 3.5 v2Anthropic55.82026-04
DeepSeek R1DeepSeek55.82026-04
DeepSeek ReasonerDeepSeek55.82026-04
Claude Haiku 4.5Anthropic54.82026-04
Claude Haiku 4.5 (latest)Anthropic54.82026-04
DeepSeek V3.1DeepSeek54.52026-04
Gemini 2.5 FlashGoogle DeepMind53.82026-04
Gemini 2.5 Flash ImageGoogle DeepMind53.82026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind53.82026-04
Gemini 2.5 Flash LiteGoogle DeepMind53.82026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind53.82026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind53.82026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind53.82026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind53.82026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind53.82026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind53.82026-04
Grok 3xAI53.52026-04
Grok 3 FastxAI53.52026-04
Grok 3 Fast LatestxAI53.52026-04
Grok 3 LatestxAI53.52026-04
Grok 3 MinixAI53.52026-04
Grok 3 Mini FastxAI53.52026-04
Grok 3 Mini Fast LatestxAI53.52026-04
Grok 3 Mini LatestxAI53.52026-04
Qwen3 32BAlibaba / Qwen Team53.52026-04
Qwen3 32B AWQAlibaba / Qwen Team53.52026-04
Qwen3 32B NVFP4NVIDIA53.52026-04
Mistral Large (latest)Mistral AI52.82026-04
Mistral Large 2.1Mistral AI52.82026-04
Mistral Large 3Mistral AI52.82026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team52.82026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team52.82026-04
DeepSeek V3 0324DeepSeek52.22026-04
GPT-4OpenAI50.82026-04
GPT-4.1 miniOpenAI50.82026-04
GPT-4.1 nanoOpenAI50.82026-04
Claude Opus 3Anthropic50.52026-04
DeepSeek ChatDeepSeek50.52026-04
DeepSeek V3DeepSeek50.52026-04
Llama 4 Maverick 17B 128E InstructMeta49.82026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta49.82026-04
Qwen2.5 72B InstructAlibaba / Qwen Team49.52026-04
Codestral (latest)Mistral AI48.52026-04
GPT-4o miniOpenAI48.52026-04
Gemma 4 31BGoogle DeepMind47.52026-04
gemma 4 31B itGoogle DeepMind47.52026-04
gemma 4 31B it GGUFUnsloth47.52026-04
Gemma 4 31B IT NVFP4NVIDIA47.52026-04
Mistral Medium (latest)Mistral AI47.22026-04
Mistral Medium 3Mistral AI47.22026-04
Mistral Medium 3.1Mistral AI47.22026-04
Claude Haiku 3.5Anthropic46.82026-04
Claude Haiku 3.5 (latest)Anthropic46.82026-04
Llama 4 Scout 17B 16EMeta46.52026-04
Llama 4 Scout 17B 16E InstructMeta46.52026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta46.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA45.82026-04
Llama-3.3-70B-InstructMeta45.82026-04
phi 4Microsoft45.52026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team45.52026-04
Gemma 4 26BGoogle DeepMind45.22026-04
Llama 3.1 70BMeta43.82026-04
Llama 3.1 70B InstructMeta43.82026-04
Mistral Small (latest)Mistral AI40.52026-04
Mistral Small 24B Instruct 2501Mistral AI40.52026-04
Mistral Small 3.1 24B Instruct 2503Mistral AI40.52026-04
Mistral Small 3.2Mistral AI40.52026-04
Mistral Small 3.2 24B Instruct 2506Mistral AI40.52026-04
Mistral Small 4Mistral AI40.52026-04
Mistral Small 4 119B 2603Mistral AI40.52026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team38.52026-04
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team38.52026-04
Phi 4 mini instructMicrosoft35.22026-04
CodeLlama 34B Instruct hfMeta30.52026-04

Data

This page as JSON · Edit on GitHub