{
 "body": "\n## What it measures\n\nThis id is lm-eval's CNN/DailyMail summarization task. The model is given a news article and must write a short abstractive summary. The reference is the article's highlight bullets, concatenated. Language is English. The Hub config is `abisee/cnn_dailymail` version `3.0.0`, the non-anonymized summarization tables associated with Abigail See's processing code.\n\nHermann et al. (2015) first released CNN and Daily Mail articles as cloze reading-comprehension data with anonymized entities. See, Liu, and Manning (ACL 2017) used a non-anonymized highlight-summary form for pointer-generator summarization. lm-eval's YAML loads that later form. The task README still leads with the 2015 paper, which is the corpus origin, not this prompt.\n\n## How it is scored\n\nlm-eval generates up to 128 tokens, then `utils.process_results` computes ROUGE-1/2/L F-measures with stemming and BERTScore precision, recall, and F1. BERTScore is wired to `distilbert-base-uncased`. There is no single official headline; reports usually quote ROUGE-2 or ROUGE-L. Zhong et al.'s 44.41 ROUGE-1 on the Hub card is an extractive system, not this zero-shot prompt.\n\nA 128-token cap is shorter than some highlight concatenations. Truncation can hurt ROUGE even when the model is on topic.\n\n## Dataset and licence\n\nTest has 11,490 examples; train 287,113; validation 13,368. The Hub card states Apache-2.0 in the YAML licence field, and the licensing section names version 1.0.0. News-article copyright beyond that card statement was not independently checked. Answers (highlights) are public.\n\nCNN articles in the original collection run April 2007\u2013April 2015; Daily Mail June 2010\u2013April 2015. Mean article length on the card is 781 tokens; highlights 56.\n\n## Who publishes it\n\nKarl Moritz Hermann and DeepMind coauthors collected the articles. Abigail See (Stanford), Peter J. Liu (Google Brain), and Christopher D. Manning published the 2017 pointer-generator paper and the non-anonymized processing repo `abisee/cnn-dailymail`. EleutherAI's harness adds the YAML task. Papers with Code still lists a document-summarization board for the corpus; that board was not scraped here.\n\n## Lineage\n\nHermann 2015 is the cloze/QA origin. Nallapati and others recast highlights as summaries. See et al. 2017 popularized the non-anonymized abstractive split (v3.0.0). This repository has [xlsum](xlsum.md) for multilingual BBC lead summarization and [legal_summarization](legal_summarization.md) for HELM legal ROUGE. Neither is CNN/DailyMail. There is no separate `cnn_dailymail` page; this assigned id is the harness task key.\n\nThe Hub card's \"Paper\" line once points at ACL anthology K16-1028 for the pointer-generator work. The citation block on the same card correctly uses P17-1099 / See et al. 2017. This page follows the citation block and arXiv 1704.04368.\n\n## Saturation and contamination\n\nNews summarization on CNN/DailyMail is a mature ROUGE task. Extractive systems already sat in the mid-40s ROUGE-1 years ago. For this exact lm-eval prompt, no current model table was opened, so no lm-eval top score is stored. Status is watch: the corpus is old, public, and heavily trained on.\n\nContamination risk is high. Many language-model corpora include these articles and highlights.\n\n## How to run it\n\n```bash\nlm-eval --tasks cnn_dailymail_abisee\n```\n\nThat string is the YAML `task:` field. The folder name and the README's task list say `cnn_dailymail`. If a runner only indexes directory names, confirm that `cnn_dailymail_abisee` is registered. The Hub path in YAML is `abisee/cnn_dailymail`, not `cnn_dailymail`.\n\nROUGE implementations and BERTScore encoders change numbers. This file uses `rouge_score` with stemming and DistilBERT. A paper that used `roberta-large` BERTScore is not this task.\n\n## Reading the numbers\n\nA strong ROUGE-2 here means the 128-token summary overlapped the highlights on bigrams. It does not mean the summary is faithful; CNN/DailyMail ROUGE is known to reward lead bias. It does not mean the model can summarize other domains; see [xlsum](xlsum.md) or [legal_summarization](legal_summarization.md). Quote ROUGE-1/2/L and BERTScore together, and name version 3.0.0. If someone reports `cnn_dailymail` without `_abisee`, check whether they ran this YAML or another copy of the corpus.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "cnn_dailymail_abisee",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "cnn_dailymail",
   "CNN/DailyMail",
   "CNN-DM",
   "abisee/cnn_dailymail"
  ],
  "category": "generation",
  "contamination": {
   "note": "CNN/DailyMail has been a standard pretraining and distillation corpus for years. Highlights and articles are public. Version 3.0.0 does not hold answers out.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0 (Hub card; licensing section names version 1.0.0)",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 11490,
   "size_note": "lm-eval uses the test split of config 3.0.0: 11,490 articles. Full 3.0.0 tables on the Hub card and datasets-server: train 287,113, validation 13,368, test 11,490 (311,971). Configs 1.0.0 and 2.0.0 list the same split sizes on this card. Articles average 781 tokens and highlights 56 tokens on the card.\n",
   "splits": "train 287,113 / validation 13,368 / test 11,490 (config 3.0.0); lm-eval test_split is test",
   "url": "https://huggingface.co/datasets/abisee/cnn_dailymail"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-031 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-031"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "cnn_dailymail_abisee",
   "opencompass": "",
   "other": ""
  },
  "id": "cnn_dailymail_abisee",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "cnn_dailymail_abisee is the runnable lm-evaluation-harness task that loads Hugging Face abisee/cnn_dailymail config 3.0.0. The model reads a CNN or Daily Mail article and writes a short multi-sentence summary. References are the journalist-written highlight bullets. Version 3.0.0 is the non-anonymized summarization split associated with See, Liu, and Manning's pointer-generator work, not Hermann et al.'s original cloze reading-comprehension dump.\n",
  "metric": {
   "baseline_note": "No random baseline. The Hub card cites Zhong et al. (2020) ROUGE-1 44.41 for an extractive system on this corpus; that is not this lm-eval prompt. BERTScore in utils.py uses distilbert-base-uncased, which is a cheaper encoder than the roberta-large setting the comments mention as optional.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "ROUGE-1/2/L F-measure and BERTScore P/R/F1",
   "random_baseline": null,
   "unit": ""
  },
  "name": "CNN/DailyMail (lm-eval, See et al. v3.0.0)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1704.04368",
   "title": "Get To The Point: Summarization with Pointer-Generator Networks",
   "url": "https://arxiv.org/abs/1704.04368",
   "year": 2017
  },
  "publisher": {
   "authors": [
    "Karl Moritz Hermann",
    "Tom\u00e1\u0161 Ko\u010disk\u00fd",
    "Edward Grefenstette",
    "Abigail See",
    "Peter J. Liu",
    "Christopher D. Manning"
   ],
   "org": "DeepMind (original collection); Stanford / Google Brain (See et al. non-anonymized summarization code)",
   "url": "https://github.com/abisee/cnn-dailymail"
  },
  "released": "2017",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/cnn_dailymail",
  "saturation": {
   "as_of": "",
   "note": "Classic extractive ROUGE-1 around the mid-40s is cited on the Hub card (Zhong et al. 2020, 44.41). LLM summarization on this prompt was not read from a current leaderboard. The task is old and widely trained on, so ROUGE gaps among strong models may be small even if this YAML has no posted ceiling.\n",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-eval cnn_dailymail.yaml (task: cnn_dailymail_abisee)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/cnn_dailymail/cnn_dailymail.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval CNN-DailyMail README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/cnn_dailymail/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval CNN/DailyMail metrics utils",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/cnn_dailymail/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face abisee/cnn_dailymail",
    "url": "https://huggingface.co/datasets/abisee/cnn_dailymail"
   },
   {
    "accessed": "2026-09-08",
    "title": "abisee/cnn_dailymail dataset card",
    "url": "https://huggingface.co/datasets/abisee/cnn_dailymail/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server split counts",
    "url": "https://datasets-server.huggingface.co/info?dataset=abisee/cnn_dailymail"
   },
   {
    "accessed": "2026-09-08",
    "title": "See, Liu, Manning 2017 pointer-generator paper",
    "url": "https://arxiv.org/abs/1704.04368"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hermann et al. 2015 original CNN/DailyMail QA paper",
    "url": "https://arxiv.org/abs/1506.03340"
   }
  ],
  "status": "active",
  "subcategory": "English news abstractive summarization (non-anonymized CNN/DailyMail v3.0.0)",
  "summary": "lm-eval zero-shot abstractive summarization of CNN/DailyMail articles (See et al. version 3.0.0), scored with ROUGE-1/2/L and BERTScore.",
  "tags": [
   "summarization",
   "generation",
   "rouge",
   "lm-eval",
   "news"
  ],
  "task_format": "Zero-shot generation. Prompt: \"Summarize the following article:\" then the article then \"Summary:\". num_fewshot 0, max_gen_toks 128, do_sample false. YAML task field is cnn_dailymail_abisee; the directory is cnn_dailymail. The task README's \"Tasks\" list names cnn_dailymail, which does not match the YAML task key.\n"
 }
}