{
 "body": "\nPart of the [MMLU](mmlu.md) family.\n\n## What it measures\n\nThis id is not one of MMLU's 57 dataset subjects. It is the \"biology\" subcategory the original\nauthors define in the benchmark repository's `categories.py`, which groups the College Biology and\nHigh School Biology subjects under one STEM label. College Biology covers molecular, cellular and\norganismal biology at undergraduate level; High School Biology covers standard secondary-school\nbiology (ecology, genetics, cell biology). Some publishers report MMLU results at this coarser\ngrain instead of, or alongside, the two subjects individually, which is what this id captures.\n\n## Reading the numbers\n\nTreat a score reported under this id as covering both College Biology and High School Biology, not\neither one alone, and compare it to the two individual subject pages (`mmlu_college_biology`,\n`mmlu_high_school_biology`) when both are available for the same model. Combined, the two subjects'\nHugging Face test splits hold 454 questions; how a publisher weights or averages the two subjects\ninto this single number is not documented and is not established here, so a \"biology\" score from\none source and one from another may not be computed the same way. See the\n[MMLU](mmlu.md) family page for the shared scoring protocol, saturation and contamination notes.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mmlu_biology",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 86.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1",
   "model_id": "deepseek/deepseek-r1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528",
   "model_id": "deepseek/deepseek-r1-0528",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 NVFP4 v2",
   "model_id": "nvidia/deepseek-r1-0528-nvfp4-v2",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 0528 Qwen3 8B",
   "model_id": "deepseek/deepseek-r1-0528-qwen3-8b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 70B",
   "model_id": "deepseek/deepseek-r1-distill-llama-70b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Llama 8B",
   "model_id": "deepseek/deepseek-r1-distill-llama-8b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 1.5B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-1-5b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "bigcode-leaderboard, provider-reports, open-llm-leaderboard, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 14B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-14b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 32B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-32b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek R1 Distill Qwen 7B",
   "model_id": "deepseek/deepseek-r1-distill-qwen-7b",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Reasoner",
   "model_id": "deepseek/deepseek-reasoner",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen 3 235B Instruct",
   "model_id": "cerebras/qwen-3-235b-a22b-instruct-2507",
   "provider": "cerebras",
   "provider_display": "Cerebras",
   "score": 83.8,
   "source": "bigcode-leaderboard, provider-reports, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen3 235B-A22B",
   "model_id": "qwen/qwen3-235b-a22b",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 83.8,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B",
   "model_id": "google/gemma-4-31b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it",
   "model_id": "google/gemma-4-31b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "gemma 4 31B it GGUF",
   "model_id": "unsloth/gemma-4-31b-it-gguf",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 31B IT NVFP4",
   "model_id": "nvidia/gemma-4-31b-it-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 81.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemma 4 26B",
   "model_id": "google/gemma-4-26b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 80.8,
   "source": "lmarena.ai, provider-reports, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 80.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 79.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 79.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "phi 4",
   "model_id": "microsoft/phi-4",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2,, domain-evals llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 mini instruct",
   "model_id": "microsoft/phi-4-mini-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, preference-evals, llm-stats, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Phi 4 multimodal instruct",
   "model_id": "microsoft/phi-4-multimodal-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 77.8,
   "source": "lmarena.ai, provider-reports, preference-evals, open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "category": "knowledge",
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 454,
   "size_note": "Not a single dataset split. Sum of the two pooled subjects' test splits in the Hugging Face parquet mirror of cais/mmlu: College Biology (144 test, 16 validation, 5 dev) plus High School Biology (310 test, 32 validation, 5 dev) = 454 test questions combined. The two subjects are separately downloadable configs; there is no single \"biology\" config in the dataset itself.\n",
   "splits": "College Biology (dev 5, validation 16, test 144) + High School Biology (dev 5, validation 32, test 310)",
   "url": "https://huggingface.co/datasets/cais/mmlu"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice A"
  },
  "harness": {
   "other": "Not a distinct lm-evaluation-harness, HELM or OpenCompass task; those harnesses run College Biology and High School Biology as separate tasks (mmlu_college_biology, mmlu_high_school_biology). This id reflects publisher-reported subcategory scores, not a harness task name."
  },
  "id": "mmlu_biology",
  "leaderboard_url": "https://github.com/hendrycks/test",
  "lineage": {
   "family": "mmlu",
   "variants": [
    "mmlu_college_biology",
    "mmlu_high_school_biology"
   ]
  },
  "measures": "This id does not correspond to a single dataset config in the Hugging Face mirror of MMLU. It corresponds to the \"biology\" subcategory the benchmark's authors define in the repository's categories.py, which pools the College Biology and High School Biology subjects -- molecular, cellular and organismal biology at an undergraduate level, and standard high-school biology (ecology, genetics, cell biology) respectively. Some publishers report MMLU broken down by this kind of subcategory rather than by all 57 individual subjects; this id captures scores reported at that grain.\n",
  "metric": {
   "baseline_note": "25% is the four-option random-guess rate for each underlying question. No dedicated human baseline exists for this subcategory grouping; see the mmlu family page for the benchmark-wide human baselines.\n",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "MMLU: Biology (subcategory)",
  "page_kind": "subset",
  "paper": {
   "arxiv": "2009.03300",
   "title": "Measuring Massive Multitask Language Understanding",
   "url": "https://arxiv.org/abs/2009.03300",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Dan Hendrycks",
    "Collin Burns",
    "Steven Basart",
    "Andy Zou",
    "Mantas Mazeika",
    "Dawn Song",
    "Jacob Steinhardt"
   ],
   "org": "UC Berkeley (original); Center for AI Safety (current host)",
   "url": "https://github.com/hendrycks/test"
  },
  "released": "2020-09",
  "repo_url": "https://github.com/hendrycks/test",
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Measuring Massive Multitask Language Understanding (Hendrycks et al., arXiv:2009.03300)",
    "url": "https://arxiv.org/abs/2009.03300"
   },
   {
    "accessed": "2026-09-07",
    "title": "hendrycks/test GitHub repository (MMLU reference implementation)",
    "url": "https://github.com/hendrycks/test"
   },
   {
    "accessed": "2026-09-07",
    "title": "categories.py: MMLU subject-to-subcategory mapping, hendrycks/test repository",
    "url": "https://github.com/hendrycks/test/blob/master/categories.py"
   },
   {
    "accessed": "2026-09-07",
    "title": "cais/mmlu dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/cais/mmlu"
   }
  ],
  "status": "active",
  "subcategory": "biology",
  "summary": "The biology subcategory of MMLU: a rollup of the College Biology and High School Biology subjects, used by publishers that report MMLU at a coarser grain than all 57 subjects.",
  "tags": [
   "knowledge",
   "multiple-choice",
   "mmlu-subset",
   "stem",
   "rollup"
  ],
  "task_format": "Four-option multiple-choice questions pooled from two underlying MMLU subjects (College Biology, High School Biology), graded on the single correct labelled option. How a given publisher averages the two subjects into one number -- an unweighted mean of per-subject accuracy, or a single accuracy over the pooled question set -- is not documented and not established here.\n"
 }
}