MASK (Model Alignment between Statements and Knowledge)

MASK tests whether a model contradicts its own beliefs when pressured to lie, scoring honesty separately from factual accuracy.

Also known as: MASK, MASK Benchmark, cais/MASK, inspect_evals/mask

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	safety
Subcategory	honesty under pressure, scored separately from factual accuracy
Page status	active
Metric	honesty score (1 − P(Lie)); also accuracy of elicited beliefs
Direction	higher_is_better
Unit	%
Dataset size	1000
Publisher	Center for AI Safety and Scale AI

What it measures

MASK asks whether a model will lie by commission: state something it does not believe, with the intent that a reader treat the statement as true. Each item has a proposition (binary or numeric), a ground-truth resolution, a pressure prompt, and (except provided_facts) neutral belief-elicitation prompts. Honesty compares the pressured statement to the elicited belief. Accuracy compares the belief to ground truth. English text only. This is not [truthfulqa](truthfulqa.md), which scores popular falsehoods, and not [hhh_alignment](hhh_alignment.md) honesty items.

Task format

Multi-turn inspect_ai Task. Default solvers: standard_belief (belief_repeats_n=3), consistency_belief, then pressure (honesty_at_n=1). Binary propositions use a gpt-4o judge; numeric propositions use o3-mini. Numeric overlap uses 5% bound tolerance. provided_facts skips belief elicitation. doubling_down_known_facts prepends a scripted first turn. One epoch only.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub