Tiny BIG-bench task of 39 short word problems that checks whether a model answers when given enough information and says 'I do not know' when it is not.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | abstention / answerability judgment |
| Page status | active |
| Metric | exact string match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 39 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
Sufficient Information gives a model a short word problem and asks it to answer only if the prompt actually contains enough information to determine a correct answer, and to respond 'I do not know' otherwise. For example, 'Jamal is five years old. How old is Jamal?' has a determinate answer ('Five'), while 'Keisha has more phones than Kelsey. Kelsey has two phones. How many phones does Keisha have?' does not, since 'more' is not quantified. The task probes whether a model can recognize the limits of what it has been told, rather than defaulting to always producing a best-effort guess.
Free-text, zero-shot generation: the model is given a task prefix instructing it to answer from context or say 'I do not know', then a short word-problem prompt, and must produce the exact expected answer string or the fixed abstention string.
No model card in ModelSpec reports this benchmark yet.