520 graduate-level condensed matter physics calculation problems scored by a symbolic partial-credit metric; the best model reached only 36 average SEED score and 28% accuracy.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | graduate-level condensed matter physics calculation problems |
| Page status | active |
| Metric | SEED score (Scalable Expression Edit Distance, fine-grained partial credit) and a separate binary accuracy |
| Direction | higher_is_better |
| Unit | SEED score (partial-credit scale); accuracy % |
| Dataset size | 520 |
| Dataset licence | Apache-2.0 (confirmed on both the GitHub repository and the Hugging Face dataset card) |
| Publisher | Shanghai Artificial Intelligence Laboratory, with the Institute of Physics / Beijing National Laboratory for Condensed Matter Physics (Chinese Academy of Sciences), Fudan University, Tongji University and Hong Kong Polytechnic University |
CMPhysBench tests whether a model can solve graduate-level condensed matter physics calculation problems, a subfield of physics concerned with the collective behaviour of matter in solid and liquid phases (magnetism, superconductivity, semiconductors and related phenomena). Despite the "CM" prefix, this is a physics-subfield benchmark, not a Chinese-language one: its questions and prompts are in English, though the 34-author team behind it is drawn predominantly from Chinese institutions. The benchmark deliberately restricts itself to calculation problems, requiring the model to independently derive a comprehensive, multi-step solution, rather than conceptual or multiple-choice questions, to probe problem-solving directly rather than recall. Six topics are covered: four core areas (Magnetism, Superconductivity, Strongly Correlated Systems and Semiconductors) chosen for domain representativeness, plus two broader categories, Theoretical Foundations (crystallography, plasmonics, phase transitions, condensed matter field theory) and Others (quantum mechanics, statistical physics, electrodynamics, quantum field theory).
Open-ended calculation problem; the model is prompted to act as a condensed matter physics expert, solve step by step using only the symbols given in the problem statement (no new symbols allowed), and present its final answer as a single LaTeX expression inside a \boxed{} environment.
No model card in ModelSpec reports this benchmark yet.