MMLU: High School Computer Science

MMLU subject subset: Introductory computer science and programming concepts at high-school level.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
Subcategorycomputer science
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size100
Dataset licenceMIT
PublisherUC Berkeley (original); Center for AI Safety (current host)

What it measures

Introductory computer science and programming concepts at high-school level. Questions are four-option multiple-choice, drawn from the MMLU test set's "computer science" subcategory within the benchmark's "STEM" top-level group, and are graded on the single correct labelled option.

Task format

Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Meta Llama 3 70BMeta92.02024-07
Meta Llama 3 70B InstructMeta92.02026-04
Meta Llama 3 70B InstructNous Research92.02026-04
Yi 1.5 34B Chat01.AI90.02026-04
Yi 1.5 34B Chat 16K01.AI90.02024-07
Yi 1.5 9B Chat01.AI87.02024-07
Yi 1.5 9B Chat 16K01.AI87.02024-07
Mixtral 8x22B Instruct v0.1Mistral AI86.02026-04
Nous Hermes 2 Yi 34BNous Research84.02024-07
Yi 1.5 34B 32K01.AI84.02024-07
Yi 1.5 34B01.AI83.02026-04
Yi 1.5 9B01.AI83.02024-07
Yi 9B01.AI83.02024-07
Yi 1.5 9B 32K01.AI82.02024-07
Yi 34B 200K01.AI82.02024-07
Yi 34B Chat01.AI81.02024-07
Mixtral 8x7B Instruct v0.1Mistral AI78.02026-04
Meta Llama 3 8B InstructMeta75.02024-07
Meta Llama 3 8B InstructNous Research75.02024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research75.02024-07
Yi 1.5 6B Chat01.AI75.02024-07
Phi 3 mini 4K instructMicrosoft73.02024-07
Mixtral 8x7B v0.1Mistral AI72.02026-04
Nous Hermes 2 SOLAR 10.7BNous Research71.02024-07
Hermes 2 Theta Llama 3 8BNous Research70.02024-07
Yi 1.5 6B01.AI69.02024-07
Meta Llama 3 8BMeta68.02024-07
Meta Llama 3 8BNous Research68.02024-07
gemma 7B itGoogle DeepMind67.02024-07
Phi 3 mini 128K instructMicrosoft67.02024-07
Hermes 2 Pro Llama 3 8BNous Research66.02024-07
Mistral 7B Instruct v0.2Mistral AI65.02024-07
Mistral 7B v0.3Mistral AI65.02024-07
mistral 7B v0.3 bnb 4bitUnsloth65.02024-07
Yi 6B01.AI65.02024-07
Yi 6B Chat01.AI65.02024-07
phi 2Microsoft64.02024-07
falcon 40BTII62.02024-07
deepseek coder 6.7B instructDeepSeek56.02024-07
Qwen2 1.5B InstructAlibaba / Qwen Team56.02024-07
deepseek coder 6.7B baseDeepSeek52.02024-07
deepseek llm 7B baseDeepSeek47.02024-07
deepseek llm 7B chatDeepSeek47.02024-07
Qwen2 0.5B InstructAlibaba / Qwen Team47.02024-07
chatglm2 6BZhipu AI42.02024-07
gemma 2BGoogle DeepMind42.02024-07
gemma 2B itGoogle DeepMind37.02024-07
deepseek coder 1.3B baseDeepSeek33.02024-07
deepseek coder 1.3B instructDeepSeek33.02024-07
OLMo 1B hfAllen AI31.02024-07

Data

This page as JSON · Edit on GitHub