Inspect Evals SAD-mini: 2,904 multiple-choice items in five tasks on whether a model knows it is an LLM and can place itself in training, evaluation, or deployment.
unassessed
| Category | safety |
|---|---|
| Subcategory | situational awareness of being an LLM (Inspect Evals SAD-mini multiple-choice slice) |
| Page status | active |
| Metric | SAD score (accuracy with invalid answers scored at chance) |
| Direction | higher_is_better |
| Dataset size | 2904 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Independent authors, Constellation, MIT, and Apollo Research; Inspect Evals port by UK AI Security Institute |
SAD tests whether a language model knows it is a model and can act on that knowledge. The paper's full suite has 7 categories, 16 tasks, and over 13,000 questions (the project site and Inspect README say over 12,000), including self-recognition and tasks that need model-specific facts. This id is the Inspect Evals port of SAD-mini: the five multiple-choice tasks that do not change from model to model. English text. It is not the BIG-bench [self_awareness](self_awareness.md) probes.
Five Inspect tasks, each a multiple-choice generation with temperature 0. Default shuffle_choices true (gold is A before shuffle). Optional system_variant plain / sp / sp_large and answer_assist prefilling "Answer: (". Lenient prefix match on letter or choice text; invalid format scores random chance 1/n_choices.
No model card in ModelSpec reports this benchmark yet.