NVIDIA's non-adversarial robustness suite: re-run MMLU-Pro, AGIEval MCQ, and MATH under prompt, choice-order, and seed changes, reporting accuracy plus consistency rate.
unassessed
| Category | composite |
|---|---|
| Subcategory | non-adversarial prompt, choice-order, and seed robustness on MMLU-Pro, AGIEval MCQ, and MATH |
| Page status | active |
| Metric | accuracy and consistency rate (CR) |
| Direction | higher_is_better |
| Dataset size | 19372 |
| Dataset licence | Apache-2.0 |
| Publisher | NVIDIA |
SCORE does not introduce new questions. It re-asks public items from MMLU-Pro, seven English AGIEval multiple-choice exams, and Hendrycks MATH while changing the prompt, the option order, or the sampling seed. Prompt robustness uses ten semantically similar templates. Choice-order robustness swaps the gold option through every listed slot (MMLU-Pro letters A–J; AGIEval only). Non-greedy robustness holds the prompt fixed and samples at temperature 0.7 with seeds 1–5. The skill is whether an instruct model keeps the same answer under those non-adversarial edits, not whether it withstands PromptBench-style attacks.
Zero-shot generative evaluation of instruct models. MMLU-Pro and AGIEval items are multiple-choice completions parsed for an answer letter. MATH items are free-form solutions. The paper generates up to 1024 tokens per call. lm-eval recommends --apply_chat_template. Non-greedy runs need five seeded jobs plus non_greedy_summarizer.py.
No model card in ModelSpec reports this benchmark yet.