MMLU: Computer Science (subcategory)

The computer science subcategory of MMLU: a rollup of four subjects, used by publishers that report MMLU at a coarser grain than all 57 subjects.

unassessed

What it measures

This id does not correspond to a single dataset config in the Hugging Face mirror of MMLU. It corresponds to the "computer science" subcategory the benchmark's authors define in the repository's categories.py, which pools four subjects: College Computer Science (algorithms and computability at undergraduate level), High School Computer Science (introductory programming concepts), Computer Security (cryptography and systems security) and Machine Learning. Some publishers report MMLU broken down by this kind of subcategory rather than by all 57 individual subjects; this id captures scores reported at that grain.

Task format

Four-option multiple-choice questions pooled from four underlying MMLU subjects, graded on the single correct labelled option. How a given publisher averages the four subjects into one number -- an unweighted mean of per-subject accuracy, or a single accuracy over the pooled question set -- is not documented and not established here.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub