A FAIR/Meta benchmark of 20 aggregated datasets testing whether models decline to answer questions that cannot or should not be answered confidently, finding reasoning fine-tuning makes this worse.
unassessed
| Category | safety |
|---|---|
| Subcategory | epistemic calibration -- abstaining on unanswerable, underspecified or false-premise questions |
| Page status | active |
| Metric | abstention recall (share of should-abstain items where the model abstained), plus precision, F1, and response accuracy on answerable items |
| Direction | higher_is_better |
| Unit | % |
| Dataset licence | CC BY-NC 4.0 |
| Publisher | FAIR at Meta |
AbstentionBench tests whether a model recognises when it should decline to give a confident, direct answer, rather than whether it can answer correctly. It aggregates questions across six scenarios where abstention (or at least hedging) is the appropriate response: the answer is genuinely unknown or undocumented, the question rests on a false premise, the question concerns events after the relevant knowledge cutoff (stale), the question is inherently subjective, the question's context is underspecified, or the user's intent behind the question is underspecified. Most of its 20 source datasets are existing benchmarks repurposed or filtered for these properties; three (GPQA-Abstain, GSM8K-Abstain, MMLU-Math-Abstain) are new variants the authors built by editing established math and science benchmarks to remove information needed to answer confidently. The paper's headline finding is that abstention remains an unsolved problem even for frontier models, that model scale barely helps, and that reasoning-focused fine-tuning specifically makes it worse.
Free-text question answering: a model reads a question drawn from one of the 20 source datasets or scenario variants and produces an open-ended response. Rather than checking the response against a fixed answer key, an LLM judge (Llama 3.1 8B Instruct, by default) classifies whether the response abstained or attempted a direct answer; for items that are answerable, a separate correctness check is also applied to the non-abstaining responses.
No model card in ModelSpec reports this benchmark yet.