A BIG-bench reading-comprehension task that asks a model, given a short crime story, to identify the perpetrator and explain the clues that support that deduction.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | narrative mystery reading comprehension |
| Page status | unknown |
| Metric | multiple_choice_grade (MC subtask) and rougeLsum (free-text subtask); BIG-bench also plots a normalized aggregate |
| Direction | higher_is_better |
| Dataset size | 477 |
| Dataset licence | BIG-bench repository Apache-2.0; Gutenberg stories under the Project Gutenberg License; 5minutemystery.com terms were not established (task authors said the site had no posted terms) |
| Publisher | BIG-bench contributors (task authors listed below) |
Minute Mysteries QA gives the model a short crime or mystery story and asks who did it, and why that identification follows from the clues. Stories often hide a contradiction or a physically impossible claim in a suspect's account. The authors designed it to need causal, implicit and social reasoning, not factoid lookup. English text. Two subtasks share the stories: multiple choice (pick the perpetrator from a short list) and free text (name the culprit and the supporting reasoning).
JSON BIG-bench task with two subdirectories. Multiple choice: one correct perpetrator among three to five names, scored as multiple_choice_grade. Free text: a longer reference that includes the culprit and the explanation, scored with ROUGE and BLEU. Zero-shot in the task keywords.
No model card in ModelSpec reports this benchmark yet.