MMMU

11.5K college-level, image-paired exam questions across six disciplines, testing expert knowledge that requires reading a figure, chart or diagram.

Also known as: Massive Multi-discipline Multimodal Understanding

unassessed

What it measures

MMMU pairs text questions with images such as charts, diagrams, maps, tables, music sheets and chemical structures, drawn from real college exams, quizzes and textbooks. It spans six core disciplines (Art and Design, Business, Science, Health and Medicine, Humanities and Social Science, and Tech and Engineering) across 30 subjects and 183 subfields, and 30 distinct image types. The goal is to test whether a model can combine expert-level subject knowledge with genuine reading of the accompanying image, rather than treating the image as decoration for a question answerable from text alone.

Task format

Mostly four-option multiple-choice questions, each paired with one or more images; a smaller portion are open-ended.

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub