HELM's repurposing of the climate subset of the SUMO web-claims dataset as document-to-title summarization, distinct from the original paper's claim-verification task.
unassessed
| Category | domain |
|---|---|
| Subcategory | single-document abstractive summarization of climate fact-check documents into a claim-style title |
| Page status | active |
| Metric | ROUGE-2 (consistent with HELM's other summarization scenarios); exact primary metric for this scenario not independently confirmed |
| Direction | higher_is_better |
| Unit | score |
| Dataset licence | Not established: no explicit licence statement was found in the sources opened for this page for either the SUMO repository or the climate_claims_raw.xlsx file HELM loads. |
| Publisher | Stanford CRFM (HELM scenario); underlying data from University of Zurich / TU Kaiserslautern authors (SUMO paper) |
As implemented in HELM, this task gives a model a document related to a climate-change claim (`Doc_text`) and asks it to generate a short title-style summary (`Title`), scored as single-document abstractive summarization. This is a repurposing of the underlying data: the source SUMO dataset (Mishra et al., 2020) was built for claim verification with extractive evidence -- given a claim and retrieved web evidence documents, decide whether the claim is true or false and extract supporting sentences -- not for title generation. HELM's scenario keeps only the climate-focused rows and reduces the task to document-to-title summarization, discarding the original claim-correctness labels. All text is English.
Free-text generation: given a climate-claim-related document, generate a short title-style summary.
No model card in ModelSpec reports this benchmark yet.