{
 "body": "\n## What it measures\n\nClimaQA tests whether a model holds accurate, textbook-level climate-science knowledge and can apply it across three different question formats. Its authors built ClimaGen, an LLM-driven pipeline that generates question-answer pairs from graduate-level climate-science textbooks with climate scientists reviewing the output, then used it to produce two datasets under the ClimaQA name: ClimaQA-Gold, a smaller, fully expert-annotated set, and ClimaQA-Silver, a larger, fully synthetic set built for scale rather than individual expert review. Within each, questions are further tagged by cognitive demand: Base (factual recall), Reasoning (multi-step scientific reasoning) and Hypothetical (scenario application), letting evaluators separate rote knowledge from applied understanding.\n\nThe benchmark's premise is that prior LLM-in-climate-science work lacked a framework capable of assessing both the quality and the scientific validity of model outputs at scale; ClimaGen's semi-automated, expert-in-the-loop pipeline is presented as a way to keep generating fresh questions without requiring a domain expert to write every one by hand.\n\n## How it is scored\n\nThree question formats are drawn from the same underlying question bank: multiple-choice (four options, a-d), cloze (fill in a missing word or short phrase), and free-form question answering (an open-ended response). None are scored by simple string matching; instead a separate LLM grader compares the model's answer against the gold answer under format-specific instructions -- strict equivalence for multiple-choice and cloze, and semantic closeness (allowing different wording, units or equivalent mathematical forms) for free-form answers. The paper's own results additionally report exact-match and a \"partial score\" specifically for the free-form format, since a single correct/incorrect judgement understates partial understanding on open-ended answers. The paper also evaluates retrieval-augmented generation (RAG) variants, both from the source textbook and from held-out material, alongside plain zero-shot and few-shot prompting, and finds RAG consistently outperforms continued pre-training or supervised fine-tuning as a way to raise scores.\n\n## Dataset and licence\n\nClimaQA-Gold holds 566 expert-validated questions: 245 multiple-choice (126 Base / 72 Reasoning / 47 Hypothetical), 161 free-form (54/52/55) and 160 cloze (not tiered by difficulty). ClimaQA-Silver holds 3,000 synthetic questions built the same way but without individual expert review: 1,000 multiple-choice (501/264/235), 1,000 free-form (507/241/252) and 1,000 cloze. Both counts are confirmed against the paper's own dataset-contents table. The data is released under an Apache-2.0 licence, stated consistently on the GitHub repository and the Hugging Face dataset card, with Gold and Silver as separate configs, each further split into mcq/cloze/ffq subsets and no train/validation partition in either.\n\n## Who publishes it\n\nClimaQA was built by Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen and Taylor Berg-Kirkpatrick at the University of California, San Diego, posted to arXiv in October 2024 and accepted at ICLR 2025. The authors maintain the ClimaGen generation pipeline and ClimaQA datasets at github.com/Rose-STL-Lab/genie-climaqa; no separate, continuously updated public leaderboard for ClimaQA was found.\n\n## Lineage\n\nNo predecessor or successor benchmark is named in the paper, and no other climate-specific benchmark currently has a page in this repository. It sits within a broader, growing set of scientific-domain QA benchmarks built with LLM-assisted, expert-in-the-loop generation pipelines, a construction pattern it shares in spirit (though not authorship or data) with other domain benchmarks here such as CareQA (`careqa.md`) and CARDBiomedBench (`cardbiomedbench.md`).\n\n## Saturation and contamination\n\nOn the paper's blended \"Overall\" score (MCQ, cloze and difficulty tiers combined), the best plain-generation model at release, gpt-4o with few-shot prompting and no retrieval, reached 86.39 out of 100; a retrieval-augmented variant reached 92.79. That leaves real headroom, and the gap is wider than it first appears once free-form is considered separately: even the best retrieval-augmented configuration reached only 71.88 exact match on free-form questions, and plain generation reached just 53-56%. This page reads ClimaQA as \"watch\" rather than \"open\" or \"saturated\": the MCQ/cloze-heavy composite is closing in on the upper 80s and low 90s for top models, while the harder open-ended component still leaves room to separate models. Contamination risk sits at medium: the datasets have been public without gating since roughly October 2024, about two years by this research date, and while questions are LLM-generated from textbooks rather than mined from an identifiable public corpus, no independent contamination study was found either way.\n\n## How to run it\n\nOpenCompass implements ClimaQA as six dataset configs -- `ClimaQA_Gold_mcq`, `ClimaQA_Gold_cloze`, `ClimaQA_Gold_ffq`, and the equivalent three for Silver -- each reading one format from an OpenCompass-hosted mirror and grading with `GenericLLMEvaluator` under a format-specific judge template (strict consistency for MCQ/cloze, closeness-based for free-form). This plain-generation setup does not reproduce the paper's retrieval-augmented or fine-tuned conditions, so OpenCompass-reported scores should be compared against the paper's non-RAG numbers, not its higher RAG-augmented ones. No lm-evaluation-harness, inspect_evals, HELM or BIG-bench implementation was found.\n\n## Reading the numbers\n\nA high ClimaQA score indicates a model can recall and apply graduate-level climate-science knowledge across factual, reasoning and applied-scenario questions, in a domain where getting details wrong has real stakes for anyone using the output for climate communication or analysis. Check which format and tier a score comes from: multiple-choice and cloze accuracy can look strong while free-form accuracy, graded for genuine scientific correctness rather than surface pattern-matching, lags well behind, and a retrieval-augmented score is not comparable to a plain-generation one. Because Gold and Silver differ in how carefully each item was human-checked, prefer Gold-derived scores when precision matters and treat Silver as a noisier, larger-sample signal.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "climaqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "domain",
  "contamination": {
   "note": "The dataset has been public on GitHub and Hugging Face without gating since around October 2024, about two years by this research date, and source questions are generated from graduate climate textbooks rather than mined verbatim from an easily identifiable existing public QA corpus. No canary string or access agreement was found, and no publisher statement or independent study of actual leakage into a specific model's training data was located.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0 (stated on both the GitHub repository and the Hugging Face dataset card)",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 566,
   "size_note": "ClimaQA-Gold: 566 expert-validated questions (245 MCQ: 126 Base/72 Reasoning/47 Hypothetical; 161 Freeform: 54/52/55; 160 Cloze, not tiered by difficulty). ClimaQA-Silver: 3,000 synthetic questions with the same three-format structure (1,000 MCQ: 501/264/235; 1,000 Freeform: 507/241/252; 1,000 Cloze), generated by the ClimaGen pipeline without individual expert review. Both totals are confirmed against the paper's own dataset-contents table. dataset.size above records Gold only, the expert-verified half; Silver is a separate, much larger configuration on the same Hugging Face repository.\n",
   "splits": "Gold and Silver are separate configs, each split into mcq / cloze / ffq; no train/validation split in either",
   "url": "https://huggingface.co/datasets/Rose-STL-Lab/ClimaQA"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 5, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "ClimaQA_Gold_{mcq,cloze,ffq} and ClimaQA_Silver_{mcq,cloze,ffq}",
   "other": ""
  },
  "id": "climaqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ClimaQA tests whether a model holds accurate, textbook-level climate-science knowledge and can apply it across three question formats. Its authors built ClimaGen, an LLM-driven pipeline that generates question-answer pairs from graduate climate-science textbooks with climate scientists reviewing the output, then used it to produce two datasets under the ClimaQA name: ClimaQA-Gold, a smaller expert-annotated set, and ClimaQA-Silver, a larger fully synthetic set for scale. Within each, questions are further split by cognitive demand into Base (factual recall), Reasoning (multi-step scientific reasoning) and Hypothetical (scenario application) tiers.\n",
  "metric": {
   "baseline_note": "The multiple-choice format alone has a 25% random baseline (four options), but ClimaQA's headline numbers blend MCQ, cloze and difficulty tiers into one \"Overall\" score with no single clean random baseline, and the free-form component is graded for closeness rather than exact match, so no unified random or human baseline is established here. In the paper's own results table, the best non-retrieval-augmented configuration, gpt-4o with few-shot prompting, reached 86.39 \"Overall\" with 56.25 exact-match / 0.89 partial-score on free-form questions; a retrieval-augmented gpt-4o variant reached 92.79 Overall and 71.88/0.94 on free-form, but retrieval augmentation is not part of the format OpenCompass's implementation runs.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (MCQ/cloze, LLM-graded exact-consistency) and LLM-graded closeness (free-form)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "ClimaQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2410.16701",
   "title": "ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models",
   "url": "https://arxiv.org/abs/2410.16701",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Veeramakali Vignesh Manivannan",
    "Yasaman Jafari",
    "Srikar Eranky",
    "Spencer Ho",
    "Rose Yu",
    "Duncan Watson-Parris",
    "Yian Ma",
    "Leon Bergen",
    "Taylor Berg-Kirkpatrick"
   ],
   "org": "University of California, San Diego",
   "url": "https://github.com/Rose-STL-Lab/genie-climaqa"
  },
  "released": "2024-10",
  "repo_url": "https://github.com/Rose-STL-Lab/genie-climaqa",
  "saturation": {
   "as_of": "2024-10",
   "note": "On the paper's blended MCQ/Cloze/difficulty-tier \"Overall\" score, the best plain-generation model (gpt-4o, few-shot, no retrieval) reached 86.39 out of 100, leaving real but narrowing headroom; a retrieval-augmented gpt-4o variant reached 92.79. The free-form component lags well behind on exact match (56.25 without retrieval, 71.88 with it), so the composite score can look closer to saturated than the open-ended half of the benchmark actually is. No independently maintained leaderboard with fresher scores was found.\n",
   "status": "watch",
   "top_score": 86.39
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "ClimaQA: An Automated Evaluation Framework for Climate Question Answering Models",
    "url": "https://arxiv.org/abs/2410.16701"
   },
   {
    "accessed": "2026-09-08",
    "title": "ClimaQA paper, full text (ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/2410.16701"
   },
   {
    "accessed": "2026-09-08",
    "title": "Rose-STL-Lab/ClimaQA dataset card, Hugging Face (Gold and Silver configs)",
    "url": "https://huggingface.co/datasets/Rose-STL-Lab/ClimaQA"
   },
   {
    "accessed": "2026-09-08",
    "title": "Rose-STL-Lab/genie-climaqa repository metadata (Apache-2.0 licence)",
    "url": "https://api.github.com/repos/Rose-STL-Lab/genie-climaqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ClimaQA_Gold dataset config (mcq/cloze/ffq grading templates)",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/ClimaQA/ClimaQA_Gold_llm_judge_gen_f15343.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ClimaQA_Silver dataset config",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/configs/datasets/ClimaQA/ClimaQA_Silver_llm_judge_gen_f15343.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass ClimaQADataset loader",
    "url": "https://github.com/open-compass/opencompass/blob/main/opencompass/datasets/climaqa.py"
   }
  ],
  "status": "active",
  "subcategory": "climate science question answering",
  "summary": "An automated framework producing 566 expert-verified (Gold) and 3,000 synthetic (Silver) climate-science QA items across multiple-choice, cloze and free-form formats.",
  "tags": [
   "domain",
   "climate-science",
   "question-answering",
   "llm-judge",
   "multi-format"
  ],
  "task_format": "Three question formats drawn from the same underlying question bank: multiple-choice (four options, a-d), cloze (fill in a missing word or short phrase), and free-form question answering (an open-ended response with no options). All three are scored by having a separate LLM grader compare the model's answer against the gold answer under format-specific grading instructions, rather than by simple string matching.\n"
 }
}