HELM's single-document summarization scenario, which scores models on abstracting BBC news articles (XSum) or CNN/DailyMail articles into short summaries, mainly by ROUGE-2.
unassessed
| Category | generation |
|---|---|
| Subcategory | single-document abstractive summarization (HELM scenario wrapping XSum and CNN/DailyMail) |
| Page status | active |
| Metric | ROUGE-2 (primary metric per HELM's own scenario metadata); other ROUGE variants also reported |
| Direction | higher_is_better |
| Unit | score |
| Dataset licence | Not established for the scenario as a whole. XSum and CNN/DailyMail are each redistributed by their own original publishers under their own terms (BBC news text and CNN/Daily Mail news text respectively); this page did not independently confirm a single licence covering HELM's repackaged, pickled copies of either dataset. |
| Publisher | Stanford Center for Research on Foundation Models (CRFM) |
This is HELM's generic single-document summarization scenario: given a news article, the model must produce a short abstractive summary. The scenario class (`SummarizationScenario`) does not define one fixed dataset; it currently wraps two underlying corpora as separate configurable variants -- XSum (BBC news articles paired with a single-sentence, highly abstractive summary) and CNN/DailyMail (news articles paired with multi-sentence "highlight" summaries), plus an "xsum-sampled" variant that subsets XSum. All variants are English news text, and the model sees only the source article, with optional truncation and length-based filtering controls exposed by the scenario code.
Free-text generation: given a news article (optionally truncated to a maximum token length), produce an abstractive summary.
No model card in ModelSpec reports this benchmark yet.