AutoBencher Capabilities is a 2,377-question HELM benchmark whose math, history, science, economics and multilingual QA items were searched for and generated by a language model, not written by people.
unassessed
| Category | composite |
|---|---|
| Subcategory | automatically-searched question answering across math, history, science, economics and multilingual knowledge |
| Page status | unknown |
| Metric | Model-judged correctness (HELM's annotation_autobencher_capabilities_score: an LLM annotator compares the free-form answer to the gold answer) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2377 |
| Publisher | Stanford University |
AutoBencher Capabilities is not a fixed, human-written test; it is one output of a search procedure called AutoBencher. An "evaluator" language model (GPT-4-0125 in the paper) is pointed at a broad domain -- math, history, science, economics, or multilinguality -- and asked to propose several narrower topic descriptions within it. For each candidate topic the evaluator LM drafts a small batch of questions and answers using "privileged information" that only it sees: Wikipedia articles for the knowledge-style domains, a Python interpreter for math, and a machine-translation step for multilingual items. Every candidate topic is then scored on how difficult it is for existing models, how much it spreads model accuracies apart, and how much it reorders the standard model leaderboard relative to existing benchmarks such as MMLU (`mmlu`), and an adaptive search loop (8 rounds proposing 5 new topic descriptions each) steers subsequent rounds toward topics that score better on those properties. What ends up in the released dataset are the questions from the topics that survived this search. The privileged source material is only ever shown to the evaluator LM that writes the questions, never to the model being tested, so answering correctly requires actually knowing the fact, computing the result, or understanding the translated question -- not access to the source the question was built from.
Open-ended, free-form question answering: the model reads a question in English (or, for the multilingual subject, a question machine-translated into Hindi, Italian, Greek or Russian) and generates a short free-text answer. There are no answer choices.
No model card in ModelSpec reports this benchmark yet.