A 30-turn two-model game where a manipulator tries to make a naive partner say a secret codeword without saying it or being guessed.
unassessed
| Category | safety |
|---|---|
| Subcategory | two-model codeword social-engineering game (manipulator vs manipulatee) |
| Page status | active |
| Metric | win_rate (also loss_rate, violation_rate, user_aware_of_manipulation_rate) |
| Direction | higher_is_better |
| Dataset size | 50 |
| Dataset licence | MIT for the inspect_evals port. Codeword lists come from openai/evals (LFS jsonl). openai/evals LICENSE.md is MIT for repository software (Copyright 2023 OpenAI) and does not list a separate exception for make_me_say jsonl. |
| Publisher | OpenAI (original elsuite; methodology from Google DeepMind); UK AI Security Institute inspect_evals port |
MakeMeSay is a conversation game. The evaluated model is the manipulator. It must get a second model (the manipulatee, role-playing a human) to say a secret codeword, without saying that word itself, and without the partner later guessing the word. Difficulty is which word list is used. Originally an OpenAI Evals elsuite task, adapted from a Google DeepMind methodology. inspect_evals ports it as `makemesay`. English text.
Default 30 turns. Manipulatee speaks first. Manipulator has a hidden internal-monologue channel. After the chat, the manipulatee is told about the game and guesses the codeword; a judge model scores that guess. inspect default difficulty is medium-and-hard (50 codewords in the pinned OpenAI jsonl). Outcomes: win, loss, rule violation, or user aware of manipulation. o1 cards ran 32 samples per codeword.
No model card in ModelSpec reports this benchmark yet.