{
 "body": "\nPart of the [MMLU](mmlu.md) family.\n\n## What it measures\n\nLicensing-exam-style questions spanning clinical, developmental and social psychology,\npsychometrics (test theory, cutoff scores, ANOVA) and professional ethics (competence,\nconfidentiality, dual relationships with clients). Like the rest of MMLU, each question gives four\nlabelled options and the model is graded on picking the single correct one, typically evaluated\n5-shot. The benchmark's own categorisation places this subject in the \"psychology\" subcategory,\nwithin the \"social sciences\" group of MMLU's four broad areas (STEM, humanities, social sciences,\nand other), alongside subjects such as high school psychology and sociology.\n\n## Reading the numbers\n\nThe Hugging Face mirror of this subject holds 612 test questions (used for scoring), plus 69\nvalidation and 5 dev questions for few-shot prompting. That is more than four times the size of\nMMLU's smallest subjects (100 questions each), so single-question noise moves the reported\npercentage less than on most MMLU subjects, though small gaps between closely scoring models are\nstill not strongly meaningful. Read it against a model's overall MMLU score and against other\nsubjects in the \"social sciences\" group rather than in isolation, and see the [MMLU](mmlu.md)\nfamily page for the shared scoring protocol, saturation and contamination notes that apply here\ntoo.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mmlu_professional_psychology",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.3,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 85.3,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 84.0,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 82.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Yi 34B",
   "model_id": "nous-research/nous-hermes-2-yi-34b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 82.4,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 82.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 81.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 80.9,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 78.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 78.4,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 78.3,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 78.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 76.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 75.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 75.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 73.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 72.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 72.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 71.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 70.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 70.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 70.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 70.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 70.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 69.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 69.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 68.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 68.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 66.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 66.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 66.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 66.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 65.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 63.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 62.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 56.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 56.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 55.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 52.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 52.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "chatglm2 6B",
   "model_id": "zhipu/chatglm2-6b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 44.4,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 41.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 38.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 37.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B instruct",
   "model_id": "deepseek/deepseek-coder-6-7b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 35.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B base",
   "model_id": "deepseek/deepseek-coder-6-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 31.7,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 27.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B base",
   "model_id": "deepseek/deepseek-coder-1-3b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 24.2,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B instruct",
   "model_id": "deepseek/deepseek-coder-1-3b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 24.2,
   "source": "open-llm-leaderboard-v1"
  }
 ],
 "page": {
  "category": "knowledge",
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 612,
   "size_note": "612 test questions (used for scoring), plus 69 validation and 5 dev (few-shot prompt) questions, per the Hugging Face parquet mirror of cais/mmlu, config 'professional_psychology'.",
   "splits": "dev (5), validation (69), test (612)",
   "url": "https://huggingface.co/datasets/cais/mmlu"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 1b, slice N"
  },
  "harness": {
   "helm": "mmlu:subject=professional_psychology",
   "lm_eval": "mmlu_professional_psychology",
   "other": "hendrycksTest-professional_psychology in the pre-2024 Open LLM Leaderboard v1 harness fork"
  },
  "id": "mmlu_professional_psychology",
  "leaderboard_url": "https://github.com/hendrycks/test",
  "lineage": {
   "family": "mmlu"
  },
  "measures": "Licensing-exam-style questions spanning clinical, developmental and social psychology, psychometrics, and professional ethics. Questions are four-option multiple-choice, drawn from the MMLU test set's \"psychology\" subcategory within the benchmark's \"social sciences\" top-level group, and are graded on the single correct labelled option.",
  "metric": {
   "baseline_note": "25% is the four-option random-guess rate. No subject-specific human baseline is given by the paper for this subject; see the mmlu family page for the benchmark-wide human baselines.",
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "MMLU: Professional Psychology",
  "page_kind": "subset",
  "paper": {
   "arxiv": "2009.03300",
   "title": "Measuring Massive Multitask Language Understanding",
   "url": "https://arxiv.org/abs/2009.03300",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Dan Hendrycks",
    "Collin Burns",
    "Steven Basart",
    "Andy Zou",
    "Mantas Mazeika",
    "Dawn Song",
    "Jacob Steinhardt"
   ],
   "org": "UC Berkeley (original); Center for AI Safety (current host)",
   "url": "https://github.com/hendrycks/test"
  },
  "released": "2020-09",
  "repo_url": "https://github.com/hendrycks/test",
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Measuring Massive Multitask Language Understanding (Hendrycks et al., arXiv:2009.03300)",
    "url": "https://arxiv.org/abs/2009.03300"
   },
   {
    "accessed": "2026-09-08",
    "title": "hendrycks/test GitHub repository (MMLU reference implementation)",
    "url": "https://github.com/hendrycks/test"
   },
   {
    "accessed": "2026-09-08",
    "title": "cais/mmlu dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/cais/mmlu"
   },
   {
    "accessed": "2026-09-08",
    "title": "cais/mmlu datasets-server size API (per-subject row counts)",
    "url": "https://datasets-server.huggingface.co/size?dataset=cais/mmlu"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness mmlu_professional_psychology task config",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/mmlu/default/mmlu_professional_psychology.yaml"
   }
  ],
  "status": "active",
  "subcategory": "psychology",
  "summary": "MMLU subject subset: Licensing-exam-style questions spanning clinical, developmental and social psychology, psychometrics and professional ethics.",
  "tags": [
   "knowledge",
   "multiple-choice",
   "mmlu-subset",
   "social-sciences"
  ],
  "task_format": "Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU."
 }
}