Scores whether a RAG system covers required topics and avoids curated unsupported claims on 869 abstract questions over papers and news.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | abstract document QA for retrieval-augmented generation |
| Page status | active |
| Metric | topic F1 (T-F1); also topic precision/recall and hallucination rates |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 869 |
| Dataset licence | ODC-By-1.0 for annotations and metadata; source documents keep their own licences |
| Publisher | The Chinese University of Hong Kong, Shenzhen (School of Data Science), with Data Science Group, Huolala |
ASTRA-QA (AbSTRAct Question Answering over documents) tests whether a retrieval-augmented system can synthesise a long answer from academic papers and news rather than extract a short fact. Questions cover five types: single-document summarisation, two-way comparison, multi-way comparison, thematic enumeration, and temporal reasoning. Each item is also run under three retrieval scopes (Simple, Middle, Hard) that grow the distractor pool while keeping the same topic-set reference. Scoring checks topic coverage against a curated answer set and matches against a curated hallucination set of plausible but unsupported topics.
Given a question and a retrieval corpus, the system returns a free-form abstractive answer. An LLM extractor (GPT-5.1 in the paper) turns that answer into a topic set, which is matched to the gold answer topics and the hallucination set.
No model card in ModelSpec reports this benchmark yet.