Processed Physics GRE exams scored as five-way multiple choice; lm-eval reports accuracy on image-free items, not Inflection's GRE percentile.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | five-way multiple-choice graduate Physics GRE items, image-free subset |
| Page status | unknown |
| Metric | acc (lm-eval loglikelihood multiple-choice) |
| Direction | higher_is_better |
| Dataset size | 75 |
| Dataset licence | MIT |
| Publisher | Inflection AI (processed exam files); Hugging Face mirror by shayekh; lm-eval task packaging by EleutherAI |
Physics GRE, in this id, is Inflection AI's processed copy of released Physics GRE forms, a graduate-school subject exam. Each item is a five-way (A-E) English physics question. Fields are `input` (stem with options inlined), `target_scores` (the correct letter scored 1), and `has_image`. Inflection scores only items without a diagram. The skill is whatever those released graduate entrance forms ask, as text-only multiple choice, not open-ended derivation and not high-school word problems.
Multiple choice over letters A-E. Inflection sampled generations and took majority vote (maj@8 / maj@32), then mapped a GRE-style raw score to a percentile. lm-evaluation-harness instead ranks the five option letters by loglikelihood (`output_type: multiple_choice`) and reports mean `acc`. There is no `_generate_until` variant in that harness directory.
No model card in ModelSpec reports this benchmark yet.