A 70-item BIG-bench task that asks whether a second English sentence follows from the first by meaning alone, not by world facts.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench two-way analytic vs non-analytic sentence entailment (70 items) |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 70 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
analytic_entailment presents two English sentences and asks whether the second is analytically entailed by the first: it must be true given what the words mean, not given extra empirical facts. Authors Rachel Etta Rudolph, Ethan Jerzak, and Alexander W. Kocurek wrote the pairs to separate meaning from regularity (a king is a man; a US president need not be). Logical words such as negation appear, so dictionary lookup of one noun is not enough. It is not NLI from MultiNLI or [anli](anli.md).
Two-option multiple choice (entailment vs no-entailment), preferred metric multiple_choice_grade. task_prefix asks whether the pair embodies an entailment relation. example_input_prefix is "Sentences:"; example_output_prefix is "Relation:". append_choices_to_input is false. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.