An automated framework producing 566 expert-verified (Gold) and 3,000 synthetic (Silver) climate-science QA items across multiple-choice, cloze and free-form formats.
unassessed
| Category | domain |
|---|---|
| Subcategory | climate science question answering |
| Page status | active |
| Metric | accuracy (MCQ/cloze, LLM-graded exact-consistency) and LLM-graded closeness (free-form) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 566 |
| Dataset licence | Apache-2.0 (stated on both the GitHub repository and the Hugging Face dataset card) |
| Publisher | University of California, San Diego |
ClimaQA tests whether a model holds accurate, textbook-level climate-science knowledge and can apply it across three question formats. Its authors built ClimaGen, an LLM-driven pipeline that generates question-answer pairs from graduate climate-science textbooks with climate scientists reviewing the output, then used it to produce two datasets under the ClimaQA name: ClimaQA-Gold, a smaller expert-annotated set, and ClimaQA-Silver, a larger fully synthetic set for scale. Within each, questions are further split by cognitive demand into Base (factual recall), Reasoning (multi-step scientific reasoning) and Hypothetical (scenario application) tiers.
Three question formats drawn from the same underlying question bank: multiple-choice (four options, a-d), cloze (fill in a missing word or short phrase), and free-form question answering (an open-ended response with no options). All three are scored by having a separate LLM grader compare the model's answer against the gold answer under format-specific grading instructions, rather than by simple string matching.
No model card in ModelSpec reports this benchmark yet.