EleutherAI lm-eval packaging of Inverse Scaling Prize classification tasks, where larger models were observed to get worse, not better.
unassessed
| Category | composite |
|---|---|
| Subcategory | contest suite of tasks where larger LMs scored worse (classification accuracy) |
| Page status | unknown |
| Metric | accuracy (acc; acc_norm also reported) |
| Direction | higher_is_better |
| Unit | % |
| Publisher | Inverse Scaling Prize (FAR / contest organizers); lm-eval port by h-albert-lee |
Inverse Scaling Prize tasks are short English probes built so next-token training can hurt accuracy as models scale. McKenzie et al. (2023) group winning tasks into four failure modes: repeating memorized text instead of following instructions, imitating bad training patterns, solving an easy distractor instead of the hard task, and overfitting to misleading few-shot demos. lm-eval implements the multiple-choice winners (plus WinoBias anti-stereotype) as log-likelihood classification. Prompt Injection, a sequence-loss winner, is omitted. This is not [Inverse IFEval](inverseifeval.md).
Multiple-choice classification. Shared YAML _inverse_scaling_mc_yaml sets output_type multiple_choice, test_split train (Hub files ship a train split used as the eval set), doc_to_text prompt, doc_to_choice classes, doc_to_target answer_index, empty target_delimiter, and metrics acc plus acc_norm. Tasks are tagged inverse_scaling_mc. winobias_antistereotype uses its own YAML (text/classes/target, test split).
No model card in ModelSpec reports this benchmark yet.