800 university-level STEM questions that require applying a named theorem, with answers as numbers, lists, booleans or multiple-choice letters.
unassessed
| Category | math |
|---|---|
| Subcategory | university STEM theorem application |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 800 |
| Dataset licence | MIT |
| Publisher | University of Waterloo / TIGER-Lab |
TheoremQA tests whether a model can apply a STEM theorem, not whether it can do grade-school arithmetic. Domain experts wrote 800 questions over 350-odd theorems in mathematics, physics, electrical engineering and computer science, and finance. Answer forms are limited to an integer, a float, a list of numbers, a boolean, or a multiple-choice option so automatic grading is possible. 51 items also include a diagram. [MMLU-Pro](mmlu_pro.md) later reused some TheoremQA stems as a source of new questions; that is a different exam.
English generation. OpenCompass's current 5-shot config (`TheoremQA_5shot`) prompts `Problem:` / `Solution:` with five worked examples and parses the completion with `TheoremQAEvaluatorV3`. A zero-shot CoT config asks for "Therefore, the answer is ..." and one of four canonical forms.
No model card in ModelSpec reports this benchmark yet.