MMLU: Physics (unresolved key)

Disputed key on 40 frontier cards: possibly a mis-keyed MMLU-Pro Physics score, possibly a classic-MMLU STEM subcategory rollup. Neither reading is confirmed; pending a card re-key.

Also known as: mmlu_pro_physics

unassessed

What it measures

Not established. This key appears on 40 frontier model cards in this repository with no notes on its source. It may be a mis-keyed MMLU-Pro Physics category score (MMLU-Pro has a ten-option Physics category; the original 57-subject MMLU has no subject or dataset config named plain "physics"), or it may be the "physics" STEM subcategory the original MMLU authors define in categories.py, which pools four classic subjects. See "What it measures" below for the evidence on each reading and why neither is confirmed.

Task format

Not established -- it depends on which reading is correct. A ten-option, chain-of-thought format if this is an MMLU-Pro category score, or the classic four-option format if this is a categories.py subcategory rollup of four MMLU subjects.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub