BIG-bench task that generates random True/False expressions with and, or, and not, then scores whether the model prefers the correct truth value.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | programmatic Boolean-algebra evaluation |
| Page status | active |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
boolean_expressions asks a model to evaluate a short English Boolean formula such as `not True or False is `. Constants are True and False; operators are and, or, and not; parentheses are optional. The task is meant to isolate zeroth-order logic from reading comprehension. BIG-bench implements it as a Python generator (task.py), not a frozen JSON list. BIG-bench Hard keeps a separate 250-example file under the same name.
Multiple-choice over {True, False} via conditional log-probability. Default generator: seed 42, lengths 1–8, up to 100 trials per length, 3-shot context. Dummy-model logs in the task header record 428 multiple-choice queries. Preferred metric multiple_choice_grade; chance 0.5, perfect 1.0.
No model card in ModelSpec reports this benchmark yet.