BIG-Bench Hard

A 23-task suite pulled from BIG-Bench specifically because prior language models failed to beat average human raters on them.

Also known as: BBH

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryreasoning
Subcategorymulti-step reasoning suite
Page statusactive
Metricaccuracy (exact match), averaged across the 23 tasks
Directionhigher_is_better
Unit%
Dataset size6511
Dataset licenceMIT
PublisherGoogle Research

What it measures

BIG-Bench Hard bundles 23 tasks selected from the much larger BIG-Bench collection, chosen because, at the time of selection, no language model in the original BIG-Bench evaluation had outperformed the average human rater on them. The tasks span logical deduction, causal judgement, object tracking and counting, date and arithmetic reasoning, and several language-understanding puzzles such as disambiguation and sarcasm detection (snarks), so the suite is best read as a stress test of multi-step reasoning under diverse, individually simple-looking task formats rather than a single coherent skill.

Task format

A mix of multiple-choice and short free-response prompts, varying by task; most are answered directly or via chain-of-thought prompting before a final answer.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
DeepSeek R1DeepSeek68.72026-04
DeepSeek R1 0528DeepSeek68.72026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA68.72026-04
DeepSeek ReasonerDeepSeek68.72026-04
Qwen2.5 72B InstructAlibaba / Qwen Team66.32026-04
Qwen2.5 Coder 32B InstructAlibaba / Qwen Team66.32026-04
Meta Llama 3 70B InstructMeta65.52026-04
Meta Llama 3 70B InstructNous Research65.52026-04
Meta Llama 3 70BMeta64.62024-07
DeepSeek ChatDeepSeek64.12026-04
DeepSeek V2DeepSeek64.12026-04
DeepSeek V2 LiteDeepSeek64.12026-04
DeepSeek V2 Lite ChatDeepSeek64.12026-04
DeepSeek V3DeepSeek64.12026-04
DeepSeek V3 0324DeepSeek64.12026-04
DeepSeek V3.1DeepSeek64.12026-04
DeepSeek V3.2DeepSeek64.12026-04
DeepSeek V3.2 ExpDeepSeek64.12026-04
Qwen2.5 Coder 32B Instruct AWQAlibaba / Qwen Team64.02026-04
Qwen3 32BAlibaba / Qwen Team62.82026-04
Qwen3 32B AWQAlibaba / Qwen Team62.82026-04
Qwen3 32B NVFP4NVIDIA62.82026-04
Gemma 4 31BGoogle DeepMind62.32026-04
gemma 4 31B itGoogle DeepMind62.32026-04
gemma 4 31B it GGUFUnsloth62.32026-04
Gemma 4 31B IT NVFP4NVIDIA62.32026-04
Qwen2.5 Coder 14B InstructAlibaba / Qwen Team61.42026-04
Yi 1.5 34B Chat 16K01.AI61.02024-07
Yi 1.5 34B 32K01.AI60.22024-07
Gemma 4 26BGoogle DeepMind60.12026-04
Mistral Large (latest)Mistral AI58.92026-04
Mistral Large 2.1Mistral AI58.92026-04
Mistral Large 3Mistral AI58.92026-04
Qwen2.5 32B InstructAlibaba / Qwen Team58.42026-04
Qwen2.5 32B Instruct AWQAlibaba / Qwen Team58.42026-04
Llama 3.3 70B Instruct NVFP4NVIDIA58.12026-04
Llama-3.3-70B-InstructMeta58.12026-04
DeepSeek R1 Distill Llama 70BDeepSeek57.22026-04
Phi 4 mini instructMicrosoft56.92026-04
Falcon3 7B InstructTII56.32025-03
Gemma 3 27BGoogle DeepMind56.22026-04
Llama 3.1 70BMeta55.82026-04
Llama 3.1 70B InstructMeta55.82026-04
Phi 3 mini 128K instructMicrosoft55.72024-07
Yi 1.5 9B Chat01.AI55.62024-07
Yi 34B Chat01.AI55.62024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research55.42024-07
glm 4 9BZhipu AI55.32025-03
GLM 4 9B 0414Zhipu AI55.32025-03
Infinity Instruct 3M 0625 Yi 1.5 9BBAAI55.12025-03
Qwen3 14BAlibaba / Qwen Team55.12026-04
Qwen3 14B AWQAlibaba / Qwen Team55.12026-04
Qwen3 14B NVFP4NVIDIA55.12026-04
Phi 3 mini 4K instructMicrosoft55.02024-07
Llama 2 70B hfMeta54.72024-07
Qwen2 VL 7B InstructAlibaba / Qwen Team54.62025-03
Qwen2 VL 7B Instruct AWQAlibaba / Qwen Team54.62025-03
Yi 34B01.AI54.62024-07
Yi 34B 200K01.AI54.42024-07
phi 4Microsoft54.32026-04
Phi 4 multimodal instructMicrosoft54.32026-04
Nous Hermes 2 SOLAR 10.7BNous Research54.12024-07
gemma 2 27B itGoogle DeepMind52.42026-04
Hermes 2 Theta Llama 3 8BNous Research52.12024-07
granite 3.0 8B instructIBM51.92025-03
Hermes 3 Llama 3.1 8BNous Research51.82025-03
Hermes 3 Llama 3.1 8B GGUFNous Research51.82025-03
Yi 1.5 9B Chat 16K01.AI51.52024-07
Yi 1.5 9B01.AI51.42024-07
Qwen2.5 14B InstructAlibaba / Qwen Team51.22026-04
Qwen2.5 14B Instruct AWQAlibaba / Qwen Team51.22026-04
Falcon3 7B BaseTII51.02025-03
Infinity Instruct 7M Gen Llama3 1 8BBAAI50.82025-03
Hermes 2 Pro Llama 3 8BNous Research50.72024-07
Qwen3 30B A3B Instruct 2507Alibaba / Qwen Team50.52026-04
Qwen3 30B A3B NVFP4NVIDIA50.52026-04
Qwen3 30B-A3BAlibaba / Qwen Team50.52026-04
DeepSeek R1 Distill Qwen 32BDeepSeek50.12026-04
Qwen2.5 Coder 7B InstructAlibaba / Qwen Team50.02026-04
Yi 1.5 9B 32K01.AI49.62024-07
Infinity Instruct 3M 0625 Llama3 8BBAAI49.52025-03
OpenHermes 2 Mistral 7BTeknium49.52025-03
Yi 9B01.AI49.42024-07
Meta Llama 3 8B InstructMeta49.12024-07
Meta Llama 3 8B InstructNous Research49.12024-07
phi 2Microsoft48.82024-07
Gemma 3 12BGoogle DeepMind48.72026-04
OpenHermes 2.5 Mistral 7BTeknium48.72025-03
Qwen2.5 Coder 7B Instruct GPTQ Int4Alibaba / Qwen Team48.62026-04
Mixtral 8x22BMistral AI48.52026-04
Mixtral 8x22B Instruct v0.1Mistral AI48.52026-04
Qwen3 8BAlibaba / Qwen Team48.32026-04
Qwen3 8B AWQAlibaba / Qwen Team48.32026-04
Qwen3 8B BaseAlibaba / Qwen Team48.32026-04
Yi 1.5 34B01.AI48.22026-04
Yi 1.5 34B Chat01.AI48.22026-04
Yi Coder 9B01.AI48.12025-03
Yi Coder 9B Chat01.AI48.12025-03
Yi 9B 200K01.AI47.92025-03
Llama 3.1 8BCerebras47.82025-03
Ministral 8B Instruct 2410Mistral AI47.62025-03
Falcon3 3B InstructTII47.52025-03
glm 4 9B chatZhipu AI47.42025-03
Command R+Cohere47.32026-04
Mistral 7B Instruct v0.3Mistral AI47.22025-03
Qwen2.5 3B InstructAlibaba / Qwen Team46.92025-03
Falcon3 Mamba 7B InstructTII46.82025-03
Llama 3.2 3B InstructMeta46.12026-04
Meta Llama 3 8BMeta46.02024-07
Meta Llama 3 8BNous Research46.02024-07
Yi 1.5 6B Chat01.AI45.72024-07
flan t5 xlGoogle DeepMind45.42025-03
Mistral 7B v0.3Mistral AI45.22024-07
mistral 7B v0.3 bnb 4bitUnsloth45.22024-07
Yi 1.5 6B01.AI44.92024-07
Mistral 7B Instruct v0.2Mistral AI44.62024-07
glm 4 9B chat hfZhipu AI44.32025-03
Falcon3 3B BaseTII44.22025-03
Mistral 7B v0.1Mistral AI44.22024-07
Mistral NemoMistral AI44.22026-04
Mistral Nemo Base 2407Mistral AI44.22026-04
Mistral Nemo Instruct 2407Mistral AI44.22026-04
granite 3.1 2B instructIBM44.12025-03
gemma 2 2B itGoogle DeepMind44.02025-03
falcon 11BTII43.92024-07
Qwen2.5 7BAlibaba / Qwen Team43.82026-04
Qwen2.5 7B InstructAlibaba / Qwen Team43.82026-04
Qwen2.5 7B Instruct AWQAlibaba / Qwen Team43.82026-04
DeepSeek R1 Distill Qwen 14BDeepSeek43.52026-04
Hermes 3 Llama 3.2 3BNous Research43.52025-03
Yi 6B01.AI43.12024-07
Qwen2.5 1.5B InstructAlibaba / Qwen Team42.92025-03
Yi 6B 200K01.AI42.92024-07
falcon mamba 7BTII42.82025-03
falcon mamba 7B instructTII42.82025-03
falcon mamba 7B instruct Q4 K M GGUFTII42.82025-03
gemma 2 9BGoogle DeepMind42.82026-04
gemma 2 9B itGoogle DeepMind42.82026-04
OpenHermes 13BTeknium42.12025-03
glm 4 9B chat 1MZhipu AI41.82025-03
flan t5 largeGoogle DeepMind41.52025-03
Llama 2 13B hfMeta41.32024-07
Llama 2 13B hfNous Research41.32024-07
Yi 6B Chat01.AI41.32024-07
Yi 6B Chat 4bits01.AI41.32025-03
Qwen2.5 1.5BAlibaba / Qwen Team40.82025-03
Qwen2.5 1.5B Instruct AWQAlibaba / Qwen Team40.82025-03
falcon 40B instructTII40.52024-07
falcon 40BTII40.22024-07
OLMo 2 1124 7BAllen AI40.22025-03
DeepSeek R1 Distill Llama 8BDeepSeek39.82026-04
granite 3.1 8B instructIBM39.52026-04
Llama 3.1 8BMeta39.22026-04
Llama 3.1 8B InstructMeta39.22026-04
Llama 3.1 8B InstructUnsloth39.22026-04
Llama 3.1 8B Instruct FP8NVIDIA39.22026-04
Llama 3.1 8B Instruct NVFP4NVIDIA39.22026-04
Llama 3.2 3BMeta39.12026-04
stablelm zephyr 3BStability AI38.72025-03
Command RCohere38.52026-04
Qwen2 1.5B InstructAlibaba / Qwen Team38.52024-07
DeepSeek R1 Distill Qwen 7BDeepSeek38.22026-04
Nous Hermes llama 2 7BNous Research38.22024-07
OLMoE 1B 7B 0125Allen AI38.22025-03
OLMoE 1B 7B 0125 InstructAllen AI38.22025-03
Phi 3.5 mini instructMicrosoft38.12026-04
Qwen2.5 Math 1.5BAlibaba / Qwen Team37.52025-03
Falcon3 1B InstructTII37.42025-03
gemma 2 2BGoogle DeepMind37.12025-03
Mixtral 8x7BMistral AI36.82026-04
Mixtral 8x7B Instruct v0.1Mistral AI36.82026-04
Mixtral 8x7B v0.1Mistral AI36.82026-04
Falcon3 10B BaseTII36.52026-04
Falcon3 10B InstructTII36.52026-04
gemma 7B itGoogle DeepMind36.52024-07
deepseek llm 7B chatDeepSeek36.32024-07
Jamba v0.1AI21 Labs36.02025-03
Falcon3 1B BaseTII35.72025-03
flan t5 baseGoogle DeepMind35.32025-03
deepseek llm 7B baseDeepSeek35.02024-07
Llama 2 7B hfMeta35.02024-07
Llama 2 7B hfNous Research35.02024-07
stablelm 3B 4e1tStability AI35.02025-03
Llama 3.2 1B InstructMeta34.82025-03
Llama 2 7B 32K InstructTogether AI34.42025-03
deepseek moe 16B baseDeepSeek34.12025-03
LLaMA 2 7B 32KTogether AI34.02025-03
OLMoE 1B 7B 0924Allen AI33.92025-03
RedPajama INCITE 7B InstructTogether AI33.82025-03
stablelm 2 1 6BStability AI33.82025-03
gemma 2BGoogle DeepMind33.72024-07
Mistral 7B Instruct v0.1Mistral AI33.52024-07
Llama 2 13B chat hfMeta33.42024-07
GPT NeoXT Chat Base 20BTogether AI33.22025-03
Qwen2.5 0.5B InstructAlibaba / Qwen Team33.22025-03
rwkv raven 14BRWKV Foundation33.12025-03
GPT JT 6B v1Together AI33.02025-03
falcon 7BTII32.92024-07
flan t5 smallGoogle DeepMind32.82025-03
granite 3.1 1B a400m instructIBM32.82025-03

Showing the top 200 of 221.

Data

This page as JSON · Edit on GitHub