13k English temporal-commonsense candidate answers; the paper scores question-level EM/F1, while lm-eval reports pair-level acc/F1.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | temporal commonsense as per-candidate yes/no plausibility |
| Page status | unknown |
| Metric | paper: question-level EM and F1; lm-eval: pair-level acc and F1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 13225 |
| Publisher | University of Pennsylvania / Allen Institute for AI / University of Illinois |
MC-TACO (Multiple Choice TemporAl COmmonsense) gives a MultiRC context sentence, a temporal question, and one candidate answer. The system must say whether that candidate is plausible. Five properties: duration, ordering, typical time, frequency, and stationarity. More than one candidate per question can be yes. English only. This is not a single exclusive MCQ and not a cooking-TACO code benchmark.
lm-eval multiple_choice on CogComp/mc_taco. Prompt: "{sentence} Question: {question} Answer: {answer} Plausible:". Choices no/yes. Target is the binary label. validation and test splits. should_decontaminate true. --limit shuffles pairs and can drop some options of a question.
No model card in ModelSpec reports this benchmark yet.