lm-eval zero-shot abstractive summarization of CNN/DailyMail articles (See et al. version 3.0.0), scored with ROUGE-1/2/L and BERTScore.
unassessed
| Category | generation |
|---|---|
| Subcategory | English news abstractive summarization (non-anonymized CNN/DailyMail v3.0.0) |
| Page status | active |
| Metric | ROUGE-1/2/L F-measure and BERTScore P/R/F1 |
| Direction | higher_is_better |
| Dataset size | 11490 |
| Dataset licence | Apache-2.0 (Hub card; licensing section names version 1.0.0) |
| Publisher | DeepMind (original collection); Stanford / Google Brain (See et al. non-anonymized summarization code) |
cnn_dailymail_abisee is the runnable lm-evaluation-harness task that loads Hugging Face abisee/cnn_dailymail config 3.0.0. The model reads a CNN or Daily Mail article and writes a short multi-sentence summary. References are the journalist-written highlight bullets. Version 3.0.0 is the non-anonymized summarization split associated with See, Liu, and Manning's pointer-generator work, not Hermann et al.'s original cloze reading-comprehension dump.
Zero-shot generation. Prompt: "Summarize the following article:" then the article then "Summary:". num_fewshot 0, max_gen_toks 128, do_sample false. YAML task field is cnn_dailymail_abisee; the directory is cnn_dailymail. The task README's "Tasks" list names cnn_dailymail, which does not match the YAML task key.
No model card in ModelSpec reports this benchmark yet.