6,348-item binary benchmark asking whether a small edit to a factually consistent summary preserved consistency with its source document, across 10 text domains.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | binary factual-consistency detection between a document and an edited summary, across 10 domains |
| Page status | active |
| Metric | balanced accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 6348 |
| Dataset licence | Apache License 2.0 (salesforce/factualNLG repository) |
| Publisher | Salesforce Research |
SummEdits tests factual-consistency detection: given a source document and a summary of it, the model must judge whether the summary remains factually consistent with the document. Every item starts from a summary an annotator has already confirmed is fluent and consistent, then a small, "atomic" edit is applied (originally by ChatGPT) that either preserves consistency or introduces a factual error; a human annotator labels the edited version as consistent or inconsistent, discarding ambiguous ("borderline") cases. The benchmark spans 10 domains -- news, podcasts, legal bill summaries (BillSum), dialogue (SamSum), Shakespeare, scientific-paper abstracts (SciTLDR), meeting summaries (QMSum), financial earnings calls (ECTSum), sales emails, and sales calls -- so it is English text throughout but deliberately not limited to news-style summarization.
Binary classification: given a document and an edited summary, judge the summary factually consistent or inconsistent with the document.
No model card in ModelSpec reports this benchmark yet.