{
 "body": "\nPart of the [MMLU](mmlu.md) family.\n\n## What it measures\n\nComparative religion: the beliefs, practices, and history of major world religions, including Christianity, Islam, Judaism, Hinduism, and Buddhism.\n\nQuestions are four-option multiple choice, drawn largely from existing exams and quizzes rather\nthan written for the benchmark. A system is scored zero-shot or few-shot by exact match against\nthe single labelled option. The `world_religions` config in `cais/mmlu` holds 171 test questions used\nfor scoring, plus 19 held out for validation and 5 reserved as few-shot dev examples.\n\n## Reading the numbers\n\nA high score means the model recalls standard facts about the beliefs, practices, and history of major world religions; the format rewards textbook breadth over nuanced understanding of any one tradition. Hendrycks et al. report aggregate human accuracy across all 57 MMLU subjects --\nabout 90% for subject experts, about 35% for non-experts -- but no world religions-specific human\nbaseline has been published. Few publishers report every one of the 57 subject scores, so the top\nof this narrow leaderboard is often an older open-weight model run through a public harness rather\nthan today's frontier model; treat a single-subject rank as a rough signal, check it against the\noverall `mmlu` score, and once scores are high, against the harder ten-option MMLU-Pro successor.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mmlu_world_religions",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 90.6,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 90.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 90.1,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 90.1,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 89.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 88.9,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 87.7,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 87.7,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 87.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 87.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Yi 34B",
   "model_id": "nous-research/nous-hermes-2-yi-34b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 87.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 86.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 86.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 84.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 83.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 83.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 83.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 82.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 81.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 81.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 81.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 81.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 80.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 80.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 80.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 77.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 77.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 76.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 76.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 76.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 74.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 74.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 69.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 54.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "chatglm2 6B",
   "model_id": "zhipu/chatglm2-6b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 47.4,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 43.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B base",
   "model_id": "deepseek/deepseek-coder-6-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 38.6,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B instruct",
   "model_id": "deepseek/deepseek-coder-6-7b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 35.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 25.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B base",
   "model_id": "deepseek/deepseek-coder-1-3b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 21.6,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B instruct",
   "model_id": "deepseek/deepseek-coder-1-3b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 21.6,
   "source": "open-llm-leaderboard-v1"
  }
 ],
 "page": {
  "aliases": [
   "world_religions"
  ],
  "category": "knowledge",
  "contamination": {
   "note": "Test questions and gold answers have been publicly downloadable since 2020 and are widely presumed to appear in the pretraining data of most current models.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 171,
   "size_note": "171 test questions, 19 validation, 5 few-shot dev examples",
   "splits": "dev (5, few-shot prompts), validation (19), test (171, scored)",
   "url": "https://huggingface.co/datasets/cais/mmlu"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "mmlu_world_religions",
   "opencompass": "",
   "other": ""
  },
  "id": "mmlu_world_religions",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "mmlu",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Comparative religion: the beliefs, practices, and history of major world religions, including Christianity, Islam, Judaism, Hinduism, and Buddhism. Framed as four-option multiple-choice questions and scored zero-shot or few-shot by exact match against the labelled option, as one of the 57 subject subsets that make up the MMLU benchmark.",
  "metric": {
   "baseline_note": "25% expected from guessing among 4 options. The paper reports only aggregate human accuracy across all 57 MMLU subjects (about 90% for subject experts, about 35% for non-experts) -- no subject-specific human baseline has been published.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25,
   "unit": "%"
  },
  "name": "MMLU: World Religions",
  "page_kind": "subset",
  "paper": {
   "arxiv": "2009.03300",
   "title": "Measuring Massive Multitask Language Understanding",
   "url": "https://arxiv.org/abs/2009.03300",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Dan Hendrycks",
    "Collin Burns",
    "Steven Basart",
    "Andy Zou",
    "Mantas Mazeika",
    "Dawn Song",
    "Jacob Steinhardt"
   ],
   "org": "UC Berkeley",
   "url": "https://github.com/hendrycks/test"
  },
  "released": "2021",
  "repo_url": "https://github.com/hendrycks/test",
  "saturation": {
   "as_of": "2026-04",
   "note": "A harder, ten-option successor (MMLU-Pro) already exists, and few publishers report every one of the 57 MMLU subject scores, so the model on top of this narrow leaderboard is often an older open-weight model evaluated through a public harness rather than today's frontier model.",
   "status": "watch",
   "top_score": 90.6
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "Measuring Massive Multitask Language Understanding (Hendrycks et al.)",
    "url": "https://arxiv.org/abs/2009.03300"
   },
   {
    "accessed": "2026-09-07",
    "title": "hendrycks/test -- official MMLU code and data repository",
    "url": "https://github.com/hendrycks/test"
   },
   {
    "accessed": "2026-09-07",
    "title": "cais/mmlu dataset card",
    "url": "https://huggingface.co/datasets/cais/mmlu"
   },
   {
    "accessed": "2026-09-07",
    "title": "lm-evaluation-harness: mmlu task configs (task names and subject-category tags)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/mmlu/default"
   }
  ],
  "status": "active",
  "subcategory": "Humanities",
  "summary": "Accuracy on MMLU's world religions questions, one of 57 subject tests of academic and professional knowledge.",
  "tags": [
   "mmlu-subject",
   "multiple-choice",
   "humanities"
  ],
  "task_format": "Four-option multiple-choice question answering (A-D), one correct answer, zero-shot or few-shot."
 }
}