{
 "body": "\n## What it measures\n\nAutoBencher Capabilities is not a fixed, human-written test; it is one output of a search procedure called AutoBencher. An evaluator language model (GPT-4-0125 in the paper) is pointed at a broad domain -- math, history, science, economics, or multilinguality -- and proposes several narrower topic descriptions within it. For each candidate the evaluator LM drafts a batch of questions and answers using privileged information only it sees: Wikipedia articles for the knowledge-style domains, a Python interpreter for math, machine translation for multilingual items. Each candidate topic is scored on how hard it is for a panel of existing models, how much it spreads their accuracies apart, and how much it reorders the standard model ranking relative to sets such as MMLU (`mmlu`); an adaptive search loop (8 rounds proposing 5 descriptions each) steers later rounds toward topics that score better on those properties. What is released is the surviving questions, covering math, history, science, economics and multilingual factual knowledge (translated into Hindi, Italian, Greek and Russian). The privileged source material is shown only to the question-writing evaluator, never to the model being tested, so answering correctly requires actually knowing the fact or computing the result, not access to whatever source built the question.\n\n## How it is scored\n\nEach item carries one gold answer from the evaluator LM's privileged source. HELM's `autobencher_capabilities` scenario scores a free-form response with a model-judged correctness annotation (`annotation_autobencher_capabilities_score`) rather than exact-match or F1, reporting one accuracy number, zero-shot, with no held-out split. Separately, the paper's own Mechanical Turk audit of the dataset found roughly 5% of the generated (question, answer) pairs wrong (3% math/economics, 6.7% history, 7.2% science) -- comparable, the authors say, to ordinary human-constructed benchmarks -- so a slice of any model's \"incorrect\" answers here may reflect a bad gold label rather than a model mistake.\n\n## Dataset and licence\n\nThe file HELM actually loads (`xlisali1/AutoBencher-capability.json`) holds 2,377 questions: 789 math, 426 multilingual, 424 economics, 414 science, 324 history, confirmed from its parquet split metadata. This does not reconcile with the paper's rounded \"around 4000 examples,\" which totals all six experimental domains including the separate safety set rather than this one file; both figures are reported since neither source resolves the gap. Neither the GitHub repository (`XiangLi1999/AutoBencher`, or its `farzaank/AutoBencher` fork) nor the Hugging Face dataset carries a licence file or tag, so terms of reuse are not established. All text is English except the translated multilingual questions.\n\n## Who publishes it\n\nAutoBencher comes from Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang and Tatsunori Hashimoto at Stanford, posted to arXiv in July 2024 and accepted at ICLR 2025. Stanford CRFM folded the datasets into HELM as the `autobencher_capabilities` scenario and a dedicated `schema_autobencher.yaml`, though that leaderboard is not currently hosted publicly (below). Independent later Hugging Face uploads (`davidheineman/autobencher-knowledge-qa`, `-math`, AI2's `allenai/autobencher-qa-33k`) show other groups re-running the method for further, larger question sets, separate from the 2,377-question file documented here.\n\n## Lineage\n\nAutoBencher Capabilities has no predecessor id here; the paper's point of comparison is hand-built sets such as MMLU (`mmlu`), against which it reports 22% more model errors and a 27% drop in ranking correlation. Its closest relative is `autobencher_safety`, built by the same procedure and paper but aimed at refusal failures rather than knowledge gaps; the two are recorded as variants of each other rather than a family, since neither is a subset of the other. No successor benchmark by this name was found.\n\n## Saturation and contamination\n\nHELM defines a leaderboard schema for this scenario, but no live public leaderboard was found: it is absent from HELM's own homepage list of hosted suites, and the URL pattern every other HELM leaderboard uses (`crfm.stanford.edu/helm/autobencher/latest/`) 404s; neither the current HELM Capabilities nor Safety leaderboard lists it either. With no score in this repository's model cards either, status is unknown rather than guessed -- there is no scoreboard to read a ceiling from. Contamination risk is medium: items are freshly generated rather than copied from an existing corpus, but they are grounded in ordinary Wikipedia facts, the file has been downloadable since mid-to-late 2024, and its construction reused GPT-4/Gemini/Claude-class models whose successors could plausibly train on crawls that include it -- though its low download count (14, at this research) suggests limited uptake so far.\n\n## How to run it\n\nThe only confirmed harness integration is HELM's `autobencher_capabilities` scenario, loading the pinned Hugging Face file and scoring zero-shot with LLM-judged correctness. No lm-evaluation-harness, inspect_evals, OpenCompass or BIG-bench task was found. One detail for anyone re-running HELM directly: `schema_autobencher.yaml` names the *same* metric, `annotation_autobencher_capabilities_score`, as `main_name` for both the capabilities and safety run groups -- worth checking against HELM's current source rather than assuming two independently named metrics.\n\n## Reading the numbers\n\nA high score shows a model does well on questions deliberately selected, by search, for being difficult and for separating models that look similar elsewhere -- more informative about narrow, adversarially-discovered gaps than a broad capability claim. Because this is a static snapshot from mid-2024 rather than a maintained set, and roughly 1 in 20 gold answers may themselves be wrong, treat any single score with some caution. With no live leaderboard, no scores in this repository's model cards, and no way to verify the census hint's 18.0, there is no basis for calling a given number high or low relative to other models -- only relative to the roughly-100% ceiling a mostly-correct gold set implies.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "autobencher_capabilities",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "composite",
  "contamination": {
   "note": "Items are freshly generated by an evaluator LM rather than copied verbatim from a pre-existing public QA corpus, which limits (but does not rule out) direct memorisation of the exact question text. However, the underlying facts are ordinary, well-indexed Wikipedia content, the dataset has been downloadable from GitHub and Hugging Face since mid-to-late 2024, and its construction reused the same families of models (GPT-4-class, Gemini, Claude) that are later trained on web-scale crawls, so exposure to this exact file during subsequent pretraining is plausible for any model trained after its release. Uptake looks limited so far -- the Hugging Face copy shows only 14 downloads at the time of this research -- which lowers but does not remove that risk.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en",
    "hi",
    "it",
    "el",
    "ru"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 2377,
   "size_note": "2,377 questions in the single Hugging Face file HELM's autobencher_capabilities scenario actually loads (xlisali1/AutoBencher-capability.json, revision efe58dd), confirmed directly from the dataset's parquet split info and from Hugging Face's datasets-server statistics endpoint, which give an exact subject breakdown: math 789, mt (multilingual) 426, econ 424, science 414, history 324. This does not reconcile with the paper's own headline figure of \"around 4000 examples,\" which is a rounded total across all six of its experimental domains (five capability domains plus the separate safety domain) rather than a count of this specific released file, and the paper separately describes constructing capability datasets in batches of 50 examples per topic description across 8 search rounds of 5 descriptions each -- a generation-time detail that does not map cleanly onto the final released row count either. Both readings are reported here because neither source resolves the gap. The single \"train\" split is used as the entire evaluation set; HELM runs it zero-shot with no held-out training portion.\n",
   "splits": "single 'train' split, 2,377 rows total, used directly as the test set (no separate validation/test carve-out)",
   "url": "https://huggingface.co/datasets/xlisali1/AutoBencher-capability.json"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 4, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "autobencher_capabilities",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "autobencher_capabilities",
  "last_updated": "2024-11",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "autobencher_safety"
   ]
  },
  "measures": "AutoBencher Capabilities is not a fixed, human-written test; it is one output of a search procedure called AutoBencher. An \"evaluator\" language model (GPT-4-0125 in the paper) is pointed at a broad domain -- math, history, science, economics, or multilinguality -- and asked to propose several narrower topic descriptions within it. For each candidate topic the evaluator LM drafts a small batch of questions and answers using \"privileged information\" that only it sees: Wikipedia articles for the knowledge-style domains, a Python interpreter for math, and a machine-translation step for multilingual items. Every candidate topic is then scored on how difficult it is for existing models, how much it spreads model accuracies apart, and how much it reorders the standard model leaderboard relative to existing benchmarks such as MMLU (`mmlu`), and an adaptive search loop (8 rounds proposing 5 new topic descriptions each) steers subsequent rounds toward topics that score better on those properties. What ends up in the released dataset are the questions from the topics that survived this search. The privileged source material is only ever shown to the evaluator LM that writes the questions, never to the model being tested, so answering correctly requires actually knowing the fact, computing the result, or understanding the translated question -- not access to the source the question was built from.\n",
  "metric": {
   "baseline_note": "No source read for this page gives a human accuracy figure for this dataset. The closest available quality signal is the paper's own Mechanical Turk audit of the gold answers themselves: reviewers judged about 5% of AutoBencher's generated (question, answer) pairs to be wrong overall (3% for math and economics, 6.7% for history, 7.2% for science), which the authors describe as comparable to the 1-5% label-error rate reported for ordinary human-constructed benchmarks. That means a slice of any model's \"incorrect\" answers on this set is plausibly a bad gold label rather than a model error, and there is no meaningful random baseline for open-ended generation.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "Model-judged correctness (HELM's annotation_autobencher_capabilities_score: an LLM annotator compares the free-form answer to the gold answer)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "AutoBencher Capabilities",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2407.08351",
   "title": "AutoBencher: Towards Declarative Benchmark Construction",
   "url": "https://arxiv.org/abs/2407.08351",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Xiang Lisa Li",
    "Farzaan Kaiyom",
    "Evan Zheran Liu",
    "Yifan Mai",
    "Percy Liang",
    "Tatsunori Hashimoto"
   ],
   "org": "Stanford University",
   "url": "https://github.com/XiangLi1999/AutoBencher"
  },
  "released": "2024-07",
  "repo_url": "https://github.com/XiangLi1999/AutoBencher",
  "saturation": {
   "as_of": "",
   "note": "HELM's codebase defines a dedicated leaderboard schema for this scenario (schema_autobencher.yaml, grouping autobencher_capabilities with autobencher_safety), but no live site for it was found: it is absent from the list of leaderboards on HELM's own homepage (crfm.stanford.edu/helm/, which links Capabilities, Safety, Lite, Classic and over twenty other suites but not AutoBencher), the URL pattern used by every other HELM suite (crfm.stanford.edu/helm/autobencher/latest/) 404s, and neither the current HELM Capabilities nor HELM Safety leaderboard lists an AutoBencher column. This repository's census hints record a score of 18.0 for this id, but that figure could not be traced to any source that actually rendered for this research, so it is not repeated as a top_score here. No model card in this repository reports this benchmark yet (checked by grep across models/).\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "AutoBencher: Towards Declarative Benchmark Construction (arXiv abstract, authors, ICLR 2025 acceptance note)",
    "url": "https://arxiv.org/abs/2407.08351"
   },
   {
    "accessed": "2026-09-08",
    "title": "AutoBencher: Towards Declarative Benchmark Construction (ar5iv full text: method, search procedure, human evaluation, results)",
    "url": "https://ar5iv.labs.arxiv.org/abs/2407.08351"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM AutoBencherCapabilitiesScenario source (dataset path, subjects, split handling)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/autobencher_capabilities_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM schema_autobencher.yaml (run-group taxonomy: 'who: synthetic model-generated questions'; metric definition)",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/static/schema_autobencher.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "xlisali1/AutoBencher-capability.json dataset API record (revision, features, 2,377-row train split)",
    "url": "https://huggingface.co/api/datasets/xlisali1/AutoBencher-capability.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face datasets-server column statistics for the capabilities file (subject/category/language breakdown)",
    "url": "https://datasets-server.huggingface.co/statistics?dataset=xlisali1%2FAutoBencher-capability.json&config=default&split=train"
   },
   {
    "accessed": "2026-09-08",
    "title": "XiangLi1999/AutoBencher GitHub repository (official code release; no LICENSE file present)",
    "url": "https://github.com/XiangLi1999/AutoBencher"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Leaderboards homepage (full list of currently hosted leaderboard suites)",
    "url": "https://crfm.stanford.edu/helm/"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM Capabilities leaderboard, rendered (core scenario group does not include AutoBencher)",
    "url": "https://crfm.stanford.edu/helm/capabilities/latest/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Direct AutoBencher leaderboard URL on the HELM site (404, confirms no live hosted page)",
    "url": "https://crfm.stanford.edu/helm/autobencher/latest/"
   }
  ],
  "status": "unknown",
  "subcategory": "automatically-searched question answering across math, history, science, economics and multilingual knowledge",
  "summary": "AutoBencher Capabilities is a 2,377-question HELM benchmark whose math, history, science, economics and multilingual QA items were searched for and generated by a language model, not written by people.",
  "tags": [
   "knowledge",
   "math",
   "multilingual",
   "synthetic-benchmark",
   "llm-generated",
   "automated-benchmark-construction",
   "question-answering"
  ],
  "task_format": "Open-ended, free-form question answering: the model reads a question in English (or, for the multilingual subject, a question machine-translated into Hindi, Italian, Greek or Russian) and generates a short free-text answer. There are no answer choices.\n"
 }
}