8,860 four-way response-selection dialogues rewritten from Chinese high-school English listening tests, scored with R@1, R@2 and MRR.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | multi-turn English dialogue response selection with four candidates |
| Page status | unknown |
| Metric | R@1 (recall at 1 among 4 candidates); also R@2 and MRR |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 8860 |
| Dataset licence | other (EleutherAI/mutual card); the loader comments 'No license found' |
| Publisher | Westlake University and Microsoft Research Asia |
MuTual gives a short multi-turn English conversation and four candidate next turns. All four replies are on-topic; only one is logically consistent with the speakers, the setting and what was already said. The model must rank the correct reply first. Dialogues were rewritten by annotators from Chinese senior-high-school English listening exams, so the English is exam English, not spontaneous chat. A harder variant, MuTual-plus (lm-eval task `mutual_plus`), replaces one candidate with a safe fallback such as “Could you repeat that?”. This is retrieval- style response selection, not open-ended chatting and not [coqa](coqa.md).
Multiple choice over four responses. lm-eval uses output_type multiple_choice, doc_to_text the detokenised article, doc_to_choice the four options, gold letter A–D. Metrics r@1, r@2 and mrr via utils.process_results. Default YAML scores the validation split (no test_split key). should_decontaminate true.
No model card in ModelSpec reports this benchmark yet.