1,297 PhD-qualifying-exam physics problems across six subfields, needing multi-step derivation; the best model in the 2025 paper solved only 59.9% of the test set.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | PhD-qualifying-exam-level physics problem solving across six subfields |
| Page status | active |
| Metric | accuracy (SymPy symbolic equivalence plus LLM-judged answer validation) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1297 |
| Dataset licence | MIT, stated explicitly in the repository's README and confirmed via the GitHub API |
| Publisher | Yale University and New York University |
This page documents the "physics" hint that resolves to OpenCompass's `PHYSICS` dataset: Feng et al.'s 2025 benchmark of graduate, PhD-qualifying-exam-level physics problems. The name collides with an unrelated BIG-bench task also called "physics" -- a much smaller, high-school-level, multiple-choice task asking a model only to identify the correct formula for a word problem, authored by two individual contributors as part of BIG-bench's 2021-2022 crowdsourced task collection. This page is about the former; see Lineage for the latter. Feng et al.'s PHYSICS requires solving expert-annotated problems across six subfields -- classical mechanics, quantum mechanics, thermodynamics and statistical mechanics, electromagnetism, atomic physics, and optics -- each demanding multi-step mathematical derivation and reasoning to a final answer, not formula selection or fact recall. About 23% of problems include an accompanying figure.
Open-ended, free-response problem solving: the model works step by step and gives its final answer in LaTeX boxed format. Some problems bundle several sub-questions that must each be answered in order. Scoring combines SymPy-based symbolic equivalence checking with an LLM-based (originally GPT-4o) natural-language answer validator, since physics answers can be correct while differing in algebraic form.
No model card in ModelSpec reports this benchmark yet.