MMLU: Medical Genetics

Accuracy on MMLU's medical genetics questions, one of 57 subject tests of academic and professional knowledge.

Also known as: medical_genetics

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
SubcategoryOther
Page statusactive
Metricaccuracy
Directionhigher_is_better
Unit%
Dataset size100
Dataset licenceMIT
PublisherUC Berkeley

What it measures

Genetics as applied to medicine: inheritance patterns, genetic disorders, chromosomal abnormalities, and the basics of genetic testing and counselling. Framed as four-option multiple-choice questions and scored zero-shot or few-shot by exact match against the labelled option, as one of the 57 subject subsets that make up the MMLU benchmark.

Task format

Four-option multiple-choice question answering (A-D), one correct answer, zero-shot or few-shot.

Models reporting this benchmark

These figures come from the model cards, which carry one collection date per card and no per-score attribution. They are shown as reported, not as verified evidence.
ModelProviderScoreCard as of
Meta Llama 3 70BMeta92.02024-07
Meta Llama 3 70B InstructMeta92.02026-04
Meta Llama 3 70B InstructNous Research92.02026-04
Yi 34B 200K01.AI88.02024-07
Yi 1.5 34B 32K01.AI87.02024-07
Yi 34B Chat01.AI87.02024-07
Nous Hermes 2 Yi 34BNous Research86.02024-07
Yi 1.5 34B01.AI86.02026-04
Mixtral 8x22B Instruct v0.1Mistral AI84.02026-04
Yi 1.5 34B Chat01.AI84.02026-04
Yi 1.5 34B Chat 16K01.AI84.02024-07
Phi 3 mini 128K instructMicrosoft83.02024-07
Phi 3 mini 4K instructMicrosoft81.02024-07
Meta Llama 3 8B InstructMeta80.02024-07
Meta Llama 3 8B InstructNous Research80.02024-07
Nous Hermes 2 Mixtral 8x7B DPONous Research80.02024-07
Yi 1.5 9B 32K01.AI80.02024-07
Yi 1.5 9B Chat01.AI80.02024-07
Yi 1.5 9B Chat 16K01.AI80.02024-07
Hermes 2 Theta Llama 3 8BNous Research79.02024-07
Meta Llama 3 8BMeta79.02024-07
Meta Llama 3 8BNous Research79.02024-07
Yi 6B01.AI79.02024-07
Yi 6B Chat01.AI79.02024-07
Hermes 2 Pro Llama 3 8BNous Research78.02024-07
Mixtral 8x7B v0.1Mistral AI78.02026-04
Mixtral 8x7B Instruct v0.1Mistral AI77.02026-04
Yi 1.5 9B01.AI77.02024-07
Yi 1.5 6B01.AI75.02024-07
Yi 9B01.AI75.02024-07
Nous Hermes 2 SOLAR 10.7BNous Research74.02024-07
gemma 7B itGoogle DeepMind72.02024-07
Mistral 7B v0.3Mistral AI70.02024-07
mistral 7B v0.3 bnb 4bitUnsloth70.02024-07
Yi 1.5 6B Chat01.AI68.02024-07
Mistral 7B Instruct v0.2Mistral AI67.02024-07
falcon 40BTII66.02024-07
phi 2Microsoft64.02024-07
Qwen2 1.5B InstructAlibaba / Qwen Team62.02024-07
deepseek llm 7B baseDeepSeek60.02024-07
deepseek llm 7B chatDeepSeek60.02024-07
chatglm2 6BZhipu AI52.02024-07
deepseek coder 6.7B baseDeepSeek47.02024-07
gemma 2BGoogle DeepMind43.02024-07
Qwen2 0.5B InstructAlibaba / Qwen Team43.02024-07
deepseek coder 6.7B instructDeepSeek41.02024-07
gemma 2B itGoogle DeepMind39.02024-07
OLMo 1B hfAllen AI27.02024-07
deepseek coder 1.3B baseDeepSeek21.02024-07
deepseek coder 1.3B instructDeepSeek21.02024-07

Data

This page as JSON · Edit on GitHub