MMLU-SR

An MMLU variant that replaces key terms with defined dummy symbols to separate conceptual reasoning from surface pattern matching.

Also known as: MMLU-SR, MMLU Symbol Replacement, MMLU-R

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	reasoning
Subcategory	symbol-replacement stress test over 57 MMLU subjects
Page status	active
Metric	accuracy (acc)
Direction	higher_is_better
Unit	%
Dataset size	42639
Dataset licence	MIT
Publisher	MMLU-SR authors

What it measures

MMLU-SR asks the model to answer familiar MMLU questions after replacing key terms with invented symbols and definitions. It has question-only, answer-only, and question-and-answer variants across the 57 MMLU subjects. The task stays text-only and multiple choice while changing terminology.

Task format

Four-option multiple choice. The replaced term is paired with a definition in the prompt. The lm-evaluation-harness exposes aggregate, question-only, answer-only, and subject tasks; the original paper also describes a fine-tuning evaluation.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub