A BIG-bench yes/no task that asks what polarity an English implicative verb entails for a simple follow-up question.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench yes/no implicative-verb polarity (149 JSON items) |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 149 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
entailed_polarity gives an English fact that uses a one-way or two-way implicative verb, then a yes/no question about the entailed event. Kaustubh Dhole built the items from Lauri Karttunen's 2012 inventory of simple and phrasal implicatives (*SEM S12-1020). One-way implicatives license a polarity in only one direction; two-way implicatives license a polarity under both positive and negative embedding. The README claims 114 verbs and 149 examples. The skill is computing that lexical entailment, not general NLI such as [anli](anli.md) and not [analytic_entailment](analytic_entailment.md).
Two-option multiple choice (yes / no), preferred metric multiple_choice_grade. task_prefix: "Given a fact, answer the following question with a yes or a no." append_choices_to_input is false. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.