A programmatic BIG-bench probe that picks the factual news summary among candidates using PMI of the summary given the document versus the summary alone.
unassessed
| Category | generation |
|---|---|
| Subcategory | BIG-bench PMI multiple-choice probe of news-summary factuality |
| Page status | unknown |
| Metric | avg_acc (mean of cnn_dm_acc and xsum_acc) |
| Direction | higher_is_better |
| Dataset size | 594 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration); University of North Carolina |
factuality_of_summary does not ask the model to write a summary. It gives a CNN/DailyMail or XSum document and several candidate summaries, one annotated as factual and the others as nonfactual, and checks whether the model assigns higher pointwise mutual information to the factual candidate. PMI is log P(summary|document) minus log P(summary) (or, on XSum, minus log P(summary | "Summarize: ")). The skill is ranking faithfulness, not ROUGE quality. It is not [cnn_dailymail_abisee](cnn_dailymail_abisee.md) and not [bbh](bbh.md).
Programmatic BIG-bench task (task.py), not a JSON example list. Zero-shot. Preferred score avg_acc, the unweighted mean of CNN/DM accuracy and XSum accuracy. Keywords: truthfulness, summarization, multiple choice. Canary GUID embedded. max_queries in TaskMetadata is 1000.
No model card in ModelSpec reports this benchmark yet.