Fake Alignment (FINE)

FINE compares a model's open-ended safety answer with swapped multiple-choice safety options to score consistency (CS) and consistent safety (CSS).

Also known as: FINE, Fake alIgNment Evaluation, fake_safety

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	safety
Subcategory	open-ended vs multiple-choice safety consistency (CS / CSS)
Page status	active
Metric	CSS (OpenCompass headline score); also CS and open_safety_score
Direction	higher_is_better
Unit	%
Dataset size	90
Dataset licence	Apache-2.0
Publisher	Fudan University and Shanghai Artificial Intelligence Laboratory

What it measures

Fake Alignment (FINE) tests whether a model that looks safe on open-ended prompts still picks the safe option when the same question is rewritten as a two-choice item. Each stem has a Positive Option and a Negative Option. The model is queried three times: free generation, choice with Positive as A, and choice with the options reversed. "Fake alignment" here means mismatched generalisation across those formats, not Anthropic's later "alignment faking" insider-threat work. Official safety.jsonl covers five categories: Fairness, Individual Harm, Legality, Privacy, Civic Virtue.

Task format

OpenCompass FakeAlignmentDataset expands each jsonl row into three generation calls (open_generation, choice_forward, choice_reverse). ZeroRetriever, GenInferencer, system prompt "You are a helpful assistant." Evaluator FakeAlignmentJudgeEvaluator needs a judge model. Official FINE.py uses GPT-4 as the open-ended safety judge.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub