HumanEval

Tests whether a model can write a correct Python function body from its signature and docstring, checked by executing hidden unit tests.

Also known as: OpenAI HumanEval

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorycoding
Subcategoryfunction-level code generation
Page statusactive
Metricpass@1
Directionhigher_is_better
Unit%
Dataset size164
Dataset licenceMIT
PublisherOpenAI

What it measures

HumanEval gives a model a partial Python file: imports, a function signature, and an English docstring describing the required behaviour, sometimes with example input/output pairs. The model must complete the function body. Each of the 164 problems was handwritten by OpenAI so it would not already appear in code scraped from GitHub at release time. The task exercises short, self-contained programming skill rather than navigating an existing codebase, using tools, or working across multiple files.

Task format

Complete a Python function body from its signature, docstring and any starter code; graded by executing the completion against hidden unit tests (pass@k).

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
GPT-5.1OpenAI93.82026-04
GPT-5.1 ChatOpenAI93.82026-04
GPT-5.1 CodexOpenAI93.82026-04
GPT-5.1 Codex MaxOpenAI93.82026-04
GPT-5.1 Codex miniOpenAI93.82026-04
o3-proOpenAI93.52026-04
Claude Mythos PreviewAnthropic93.22026-04
Claude Opus 4Anthropic93.22026-04
Claude Opus 4.6Anthropic93.22026-04
o3OpenAI92.82026-04
o3-deep-researchOpenAI92.82026-04
Claude Opus 4.5Anthropic92.52026-04
Claude Opus 4.5 (latest)Anthropic92.52026-04
Qwen3-Coder 480B-A35B InstructAlibaba / Qwen Team92.52026-04
GPT-5OpenAI92.12026-04
GPT-5 Chat (latest)OpenAI92.12026-04
GPT-5 ProOpenAI92.12026-04
GPT-5-CodexOpenAI92.12026-04
GPT-5.2OpenAI92.12026-04
GPT-5.2 ChatOpenAI92.12026-04
GPT-5.2 CodexOpenAI92.12026-04
GPT-5.2 ProOpenAI92.12026-04
GPT-5.3 Chat (latest)OpenAI92.12026-04
GPT-5.3 CodexOpenAI92.12026-04
GPT-5.3 Codex SparkOpenAI92.12026-04
GPT-5.4OpenAI92.12026-04
GPT-5.4 miniOpenAI92.12026-04
GPT-5.4 nanoOpenAI92.12026-04
GPT-5.4 ProOpenAI92.12026-04
Claude Opus 4.1Anthropic91.72026-04
Claude Opus 4.1 (latest)Anthropic91.72026-04
Claude Sonnet 4Anthropic91.52026-04
Claude Sonnet 4.6Anthropic91.52026-04
Gemini 2.5 ProGoogle DeepMind91.22026-04
Gemini 2.5 Pro Preview 05-06Google DeepMind91.22026-04
Gemini 2.5 Pro Preview 06-05Google DeepMind91.22026-04
Gemini 2.5 Pro Preview TTSGoogle DeepMind91.22026-04
o4-miniOpenAI91.22026-04
o4-mini-deep-researchOpenAI91.22026-04
Grok 4xAI91.02026-04
Grok 4 FastxAI91.02026-04
Grok 4 Fast (Non-Reasoning)xAI91.02026-04
Grok 4.1 FastxAI91.02026-04
Grok 4.1 Fast (Non-Reasoning)xAI91.02026-04
Grok 4.20 (Non-Reasoning)xAI91.02026-04
Grok 4.20 (Reasoning)xAI91.02026-04
Grok 4.20 Multi-AgentxAI91.02026-04
Claude Sonnet 4.5Anthropic90.82026-04
Claude Sonnet 4.5 (latest)Anthropic90.82026-04
Claude Opus 4 (latest)Anthropic90.52026-04
DeepSeek R1 0528DeepSeek90.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA90.52026-04
GPT-4.1OpenAI90.22026-04
GPT-4oOpenAI90.22026-04
GPT-4o (2024-05-13)OpenAI90.22026-04
GPT-4o (2024-08-06)OpenAI90.22026-04
GPT-4o (2024-11-20)OpenAI90.22026-04
o1-proOpenAI90.22026-04
Qwen3 235B-A22BAlibaba / Qwen Team89.52026-04
DeepSeek R1DeepSeek89.22026-04
DeepSeek ReasonerDeepSeek89.22026-04
o3-miniOpenAI88.92026-04
Gemini 2.5 FlashGoogle DeepMind88.82026-04
Gemini 2.5 Flash ImageGoogle DeepMind88.82026-04
Gemini 2.5 Flash Image (Preview)Google DeepMind88.82026-04
Gemini 2.5 Flash LiteGoogle DeepMind88.82026-04
Gemini 2.5 Flash Lite Preview 06-17Google DeepMind88.82026-04
Gemini 2.5 Flash Lite Preview 09-25Google DeepMind88.82026-04
Gemini 2.5 Flash Preview 04-17Google DeepMind88.82026-04
Gemini 2.5 Flash Preview 05-20Google DeepMind88.82026-04
Gemini 2.5 Flash Preview 09-25Google DeepMind88.82026-04
Gemini 2.5 Flash Preview TTSGoogle DeepMind88.82026-04
Claude Sonnet 4 (latest)Anthropic88.72026-04
Codestral (latest)Mistral AI88.52026-04
DeepSeek V3.2DeepSeek88.52026-04
DeepSeek V3.2 ExpDeepSeek88.52026-04
GPT-5 MiniOpenAI88.52026-04
o1OpenAI88.52026-04
o1-previewOpenAI88.52026-04
Claude Sonnet 3.7Anthropic88.22026-04
GPT-4 TurboOpenAI87.82026-04
DeepSeek V3.1DeepSeek87.52026-04
GPT-4o miniOpenAI87.22026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team87.22026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team87.22026-04
Claude Sonnet 3.5Anthropic86.82026-04
Claude Sonnet 3.5 v2Anthropic86.82026-04
GPT-4.1 miniOpenAI86.52026-04
Grok 3xAI86.52026-04
Grok 3 FastxAI86.52026-04
Grok 3 Fast LatestxAI86.52026-04
Grok 3 LatestxAI86.52026-04
Magistral Medium (latest)Mistral AI86.52026-04
Claude Haiku 4.5Anthropic86.22026-04
Claude Haiku 4.5 (latest)Anthropic86.22026-04
DeepSeek ChatDeepSeek85.82026-04
DeepSeek V2DeepSeek85.82026-04
DeepSeek V2 LiteDeepSeek85.82026-04
DeepSeek V2 Lite ChatDeepSeek85.82026-04
DeepSeek V3DeepSeek85.82026-04
DeepSeek V3 0324DeepSeek85.82026-04
Gemini 2.0 FlashGoogle DeepMind85.52026-04
Llama 4 Maverick 17B 128E InstructMeta85.52026-04
Llama-4-Maverick-17B-128E-Instruct-FP8Meta85.52026-04
Qwen3 32BAlibaba / Qwen Team85.22026-04
Qwen3 32B AWQAlibaba / Qwen Team85.22026-04
Qwen3 32B NVFP4NVIDIA85.22026-04
o1-miniOpenAI85.02026-04
Claude Opus 3Anthropic84.92026-04
Qwen2.5 72B InstructAlibaba / Qwen Team84.52026-04
Mistral Large (latest)Mistral AI84.22026-04
Mistral Large 2.1Mistral AI84.22026-04
Mistral Large 3Mistral AI84.22026-04
Gemini 1.5 ProGoogle DeepMind84.12026-04
Gemma 4 31BGoogle DeepMind84.12026-04
gemma 4 31B itGoogle DeepMind84.12026-04
gemma 4 31B it GGUFUnsloth84.12026-04
Gemma 4 31B IT NVFP4NVIDIA84.12026-04
Claude Haiku 3.5Anthropic82.52026-04
Claude Haiku 3.5 (latest)Anthropic82.52026-04
Gemma 4 26BGoogle DeepMind82.52026-04
gemma 4 26B A4B itGoogle DeepMind82.52026-04
gemma 4 26B A4B it GGUFUnsloth82.52026-04
QwQ PlusAlibaba / Qwen Team82.52026-04
GPT-5 NanoOpenAI82.02026-04
Grok 3 MinixAI82.02026-04
Grok 3 Mini FastxAI82.02026-04
Grok 3 Mini Fast LatestxAI82.02026-04
Grok 3 Mini LatestxAI82.02026-04
Llama 3.1 405BMeta81.22026-04
Llama 3.1 405B FP8Meta81.22026-04
Llama 3.1 405B InstructMeta81.22026-04
Llama 3.1 405B Instruct FP8Meta81.22026-04
DeepSeek R1 Distill Llama 70BDeepSeek80.52026-04
Grok 2xAI80.52026-04
Grok 2 (1212)xAI80.52026-04
Grok 2 LatestxAI80.52026-04
Grok 2 VisionxAI80.52026-04
Grok 2 Vision (1212)xAI80.52026-04
Grok 2 Vision LatestxAI80.52026-04
Llama 3.3 70B Instruct NVFP4NVIDIA80.52026-04
Llama-3.3-70B-InstructMeta80.52026-04
Qwen3 30B A3B Instruct 2507Alibaba / Qwen Team80.52026-04
Qwen3 30B A3B NVFP4NVIDIA80.52026-04
Qwen3 30B-A3BAlibaba / Qwen Team80.52026-04
Gemini 2.0 Flash LiteGoogle DeepMind80.22026-04
Llama 4 Scout 17B 16EMeta80.22026-04
Llama 4 Scout 17B 16E InstructMeta80.22026-04
Llama-4-Scout-17B-16E-Instruct-FP8Meta80.22026-04
Gemma 3 27BGoogle DeepMind79.52026-04
Command ACohere78.52026-04
Command A ReasoningCohere78.52026-04
Gemini 1.5 FlashGoogle DeepMind78.52026-04
Gemini 1.5 Flash-8BGoogle DeepMind78.52026-04
GPT-4.1 nanoOpenAI78.52026-04
Magistral SmallMistral AI78.52026-04
Magistral Small 2506Mistral AI78.52026-04
phi 4Microsoft78.52026-04
Phi 4 multimodal instructMicrosoft78.52026-04
DeepSeek R1 Distill Qwen 32BDeepSeek78.22026-04
Qwen2.5 32B InstructAlibaba / Qwen Team78.22026-04
Qwen2.5 32B Instruct AWQAlibaba / Qwen Team78.22026-04
Qwen3 14BAlibaba / Qwen Team76.52026-04
Qwen3 14B AWQAlibaba / Qwen Team76.52026-04
Qwen3 14B NVFP4NVIDIA76.52026-04
Llama 3.1 70BMeta76.22026-04
Llama 3.1 70B InstructMeta76.22026-04
Claude Haiku 3Anthropic75.92026-04
Claude Sonnet 3Anthropic73.02026-04
Command R+Cohere72.52026-04
DeepSeek R1 Distill Qwen 14BDeepSeek72.52026-04
gemma 2 27B itGoogle DeepMind72.52026-04
Qwen2.5 14B InstructAlibaba / Qwen Team72.52026-04
Qwen2.5 14B Instruct AWQAlibaba / Qwen Team72.52026-04
Gemma 3 12BGoogle DeepMind72.02026-04
Phi 4 mini instructMicrosoft70.22026-04
Qwen3 8BAlibaba / Qwen Team70.22026-04
Qwen3 8B AWQAlibaba / Qwen Team70.22026-04
Qwen3 8B BaseAlibaba / Qwen Team70.22026-04
GPT-4OpenAI67.02026-04
Command RCohere652026-04
DeepSeek R1 Distill Qwen 7BDeepSeek652026-04
Qwen2.5 7BAlibaba / Qwen Team65.02026-04
Qwen2.5 7B InstructAlibaba / Qwen Team652026-04
Qwen2.5 7B Instruct AWQAlibaba / Qwen Team652026-04
DeepSeek R1 Distill Llama 8BDeepSeek62.52026-04
gemma 2 9BGoogle DeepMind60.52026-04
gemma 2 9B itGoogle DeepMind60.52026-04
Qwen3 4BAlibaba / Qwen Team60.52026-04
Qwen3 4B BaseAlibaba / Qwen Team60.52026-04
Qwen3 4B Instruct 2507Alibaba / Qwen Team60.52026-04
Qwen3 4B Instruct 2507 FP8Alibaba / Qwen Team60.52026-04
Gemma 3 4BGoogle DeepMind58.52026-04
gemma 3 4B ptGoogle DeepMind58.52026-04
Llama 3.1 8BMeta55.52026-04
Llama 3.1 8B InstructMeta55.52026-04
Llama 3.1 8B InstructUnsloth55.52026-04
Llama 3.1 8B Instruct FP8NVIDIA55.52026-04
Llama 3.1 8B Instruct NVFP4NVIDIA55.52026-04
GPT-3.5-turboOpenAI48.12026-04

Data

This page as JSON · Edit on GitHub