An 858-item sparse subsample of ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande that reconstructs Open LLM Leaderboard scores from a few percent of the items.
unassessed
| Category | composite |
|---|---|
| Subcategory | sparse reconstruction of Open LLM Leaderboard v1 benchmarks |
| Page status | active |
| Metric | accuracy (group mean of per-source acc; GSM8K exact match), plus optional reconstructed Open LLM Leaderboard scores |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 858 |
| Dataset licence | CC-BY-NC-SA-4.0 |
| Publisher | Human-Centered AI, Helmholtz Munich |
metabench does not add a new skill. It keeps the most informative items from six public benchmarks used on Hugging Face's Open LLM Leaderboard v1: ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande. The authors fit item-response models on thousands of leaderboard submissions, then keep a few hundred items per source so that latent-ability estimates can reconstruct the original six scores and their mean. English text. Five of the six sources are multiple choice; GSM8K is free-form arithmetic. A secondary 751-item set and choice-permuted copies exist for repeat evaluation.
Mix of multiple-choice log-likelihood (ARC, HellaSwag, MMLU, TruthfulQA, WinoGrande) and generate-until exact match (GSM8K). lm-evaluation-harness tasks bake in the original few-shot preprompts and set num_fewshot to 0 so those shots are not added twice.
No model card in ModelSpec reports this benchmark yet.