A BIG-bench task that phrases a chain of nested boolean functions as a word problem about people who tell the truth or lie, and asks the model to answer yes or no.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | multi-step boolean logic (nested truth-teller/liar word problems) |
| Page status | active |
| Metric | Multiple choice grade (accuracy on the yes/no final-truth-value question) |
| Direction | higher_is_better |
| Unit | accuracy |
| Dataset size | 1600 |
| Publisher | Google (BIG-bench collaboration); task author Roman Novak |
Web of Lies gives a model a short narrative in which several people each state whether another person tells the truth or lies, forming a chain equivalent to a composition of negation and identity functions, f_n(f_{n-1}(...f_1(x)...)). The model must determine whether the final person in the chain is telling the truth, answering yes or no. It measures multi-step boolean reasoning and the ability to track state (truth-value) through a chain of statements, framed as a naturalistic word problem rather than an explicit logic formula.
Zero-shot binary (yes/no) multiple-choice classification, scored with BIG-bench's multiple_choice_grade metric. Examples are generated procedurally at runtime by randomly assigning names and truth/lie statements to build a chain of a given length, rather than drawn from a fixed, pre-written item bank.
No model card in ModelSpec reports this benchmark yet.