MMLU: Astronomy

MMLU subject subset: Celestial mechanics, stars, planets and cosmology, at an undergraduate astronomy-course level.

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
Subcategoryphysics
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size152
Dataset licenceMIT
PublisherUC Berkeley (original); Center for AI Safety (current host)

What it measures

Celestial mechanics, stars, planets and cosmology, at an undergraduate astronomy-course level. Questions are four-option multiple-choice, drawn from the MMLU test set's "physics" subcategory within the benchmark's "STEM" top-level group, and are graded on the single correct labelled option.

Task format

Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Meta Llama 3 70BMeta93.42024-07
Meta Llama 3 70B InstructMeta93.42026-04
Meta Llama 3 70B InstructNous Research93.42026-04
Nous Hermes 2 Yi 34BNous Research89.52024-07
Yi 1.5 34B 32K01.AI89.52024-07
Yi 34B 200K01.AI89.52024-07
Yi 1.5 34B01.AI88.22026-04
Yi 1.5 34B Chat01.AI87.52026-04
Yi 1.5 34B Chat 16K01.AI87.52024-07
Mixtral 8x22B Instruct v0.1Mistral AI86.22026-04
Yi 34B Chat01.AI85.52024-07
Mixtral 8x7B v0.1Mistral AI82.92026-04
Claude Opus 4Anthropic80.52026-04
Claude Opus 4.6Anthropic80.52026-04
Nous Hermes 2 Mixtral 8x7B DPONous Research80.32024-07
Gemini 2.5 ProGoogle DeepMind79.82026-04
GPT-4.1OpenAI79.52026-04
Mixtral 8x7B Instruct v0.1Mistral AI78.92026-04
Phi 3 mini 128K instructMicrosoft78.92024-07
GPT-4oOpenAI78.22026-04
GPT-4o (2024-05-13)OpenAI78.22026-04
GPT-4o (2024-08-06)OpenAI78.22026-04
GPT-4o (2024-11-20)OpenAI78.22026-04
GPT-4o miniOpenAI78.22026-04
Claude Sonnet 4Anthropic77.82026-04
Claude Sonnet 4.5Anthropic77.82026-04
Claude Sonnet 4.5 (latest)Anthropic77.82026-04
Phi 3 mini 4K instructMicrosoft77.02024-07
Yi 1.5 9B 32K01.AI77.02024-07
DeepSeek R1DeepSeek76.52026-04
DeepSeek R1 0528DeepSeek76.52026-04
DeepSeek R1 0528 NVFP4 v2NVIDIA76.52026-04
DeepSeek R1 0528 Qwen3 8BDeepSeek76.52026-04
DeepSeek R1 Distill Llama 70BDeepSeek76.52026-04
DeepSeek R1 Distill Llama 8BDeepSeek76.52026-04
DeepSeek R1 Distill Qwen 1.5BDeepSeek76.52026-04
DeepSeek R1 Distill Qwen 14BDeepSeek76.52026-04
DeepSeek R1 Distill Qwen 32BDeepSeek76.52026-04
DeepSeek R1 Distill Qwen 7BDeepSeek76.52026-04
DeepSeek ReasonerDeepSeek76.52026-04
Nous Hermes 2 SOLAR 10.7BNous Research76.32024-07
Qwen 3 235B InstructCerebras75.82026-04
Qwen3 235B-A22BAlibaba / Qwen Team75.82026-04
Yi 1.5 9B01.AI75.72024-07
Yi 9B01.AI75.72024-07
Yi 1.5 9B Chat 16K01.AI74.32024-07
Gemma 4 31BGoogle DeepMind73.52026-04
gemma 4 31B itGoogle DeepMind73.52026-04
gemma 4 31B it GGUFUnsloth73.52026-04
Gemma 4 31B IT NVFP4NVIDIA73.52026-04
Mistral Large (latest)Mistral AI73.52026-04
Mistral Large 2.1Mistral AI73.52026-04
Mistral Large 3Mistral AI73.52026-04
gemma 7B itGoogle DeepMind73.02024-07
Yi 1.5 9B Chat01.AI72.42024-07
Gemma 4 26BGoogle DeepMind72.22026-04
Llama 3.3 70B Instruct NVFP4NVIDIA71.52026-04
Llama-3.3-70B-InstructMeta71.52026-04
Meta Llama 3 8B InstructMeta70.42024-07
Meta Llama 3 8B InstructNous Research70.42024-07
Llama 3.1 70BMeta70.22026-04
Llama 3.1 70B InstructMeta70.22026-04
Hermes 2 Pro Llama 3 8BNous Research69.72024-07
Hermes 2 Theta Llama 3 8BNous Research69.12024-07
phi 4Microsoft68.82026-04
Phi 4 mini instructMicrosoft68.82026-04
Phi 4 multimodal instructMicrosoft68.82026-04
Meta Llama 3 8BMeta67.82024-07
Meta Llama 3 8BNous Research67.82024-07
Yi 1.5 6B Chat01.AI67.12024-07
Yi 1.5 6B01.AI66.42024-07
Yi 6B01.AI65.82024-07
Yi 6B Chat01.AI65.82024-07
Mistral 7B v0.3Mistral AI65.12024-07
mistral 7B v0.3 bnb 4bitUnsloth65.12024-07
Mistral 7B Instruct v0.2Mistral AI62.52024-07
phi 2Microsoft58.62024-07
falcon 40BTII57.22024-07
Qwen2 1.5B InstructAlibaba / Qwen Team55.32024-07
chatglm2 6BZhipu AI53.32024-07
deepseek llm 7B baseDeepSeek52.62024-07
deepseek llm 7B chatDeepSeek52.62024-07
gemma 2BGoogle DeepMind42.82024-07
Qwen2 0.5B InstructAlibaba / Qwen Team41.42024-07
deepseek coder 6.7B baseDeepSeek35.52024-07
deepseek coder 6.7B instructDeepSeek33.62024-07
gemma 2B itGoogle DeepMind33.62024-07
OLMo 1B hfAllen AI20.42024-07
deepseek coder 1.3B baseDeepSeek19.72024-07
deepseek coder 1.3B instructDeepSeek19.72024-07

Data

This page as JSON · Edit on GitHub