ORT checks whether an unlearning method erases knowledge of a real person across four query forms, not only the form used in the unlearning samples.
unassessed
| Category | safety |
|---|---|
| Subcategory | form-independent LLM unlearning (forget vs retain, four query formats) |
| Page status | active |
| Metric | joint probability of the gold answer given the prompt |
| Direction | lower_is_better |
| Dataset size | 35052 |
| Publisher | Beijing University of Posts and Telecommunications, Shandong University, and Institute of Automation, Chinese Academy of Sciences |
ORT is a text-only English unlearning test. Each target is a well-known real person. After unlearning, the model is queried on a forget set about that person and on a retain set about other people. The same facts appear in four formats: simple QA, fill-in-the-blank, multiple-choice labels, and subtoken-inducing QA that forces character-level tokens. A method that only suppresses the training form can still answer the other three. The paper calls that failure Form-Dependent Bias. Utility is checked separately on MMLU, TruthfulQA, TriviaQA, and AlpacaEval rather than inside ORT itself.
Single-target unlearning, then prompt-based probes. Training corpora come in three styles (unstructured text, refusal QA, preference pairs). Evaluation prompts are QA, fill-in-the-blank, multiple-choice, or subtoken QA on both the forget set and the retain set.
No model card in ModelSpec reports this benchmark yet.