Measures how much harmful, specific and convincing content a model produces on forbidden prompts, with or without a jailbreak applied, using an LLM-judged rubric.
unassessed
| Category | safety |
|---|---|
| Subcategory | jailbreak robustness |
| Page status | active |
| Metric | StrongREJECT score |
| Direction | lower_is_better |
| Unit | points |
| Dataset size | 313 |
| Dataset licence | MIT (for the project's own code and custom-authored prompts; prompts sourced from AdvBench and DAN are MIT-licensed upstream, and prompts drawn from MasterKey, MaliciousInstruct, HarmfulQ and the OpenAI GPT-4 system card carry no formal re-licensing from those sources) |
| Publisher | UC Berkeley (Center for Human-Compatible AI) and collaborators |
StrongREJECT gives a model a curated set of forbidden prompts -- requests for genuinely harmful, specific assistance across categories such as illegal goods and services, non-violent crime, hate/harassment/discrimination, disinformation, violence and sexual content -- and records how the model responds, either directly or after a jailbreak technique has rewritten the prompt. It targets whether jailbreak attacks actually extract usable harmful content, rather than just whether a model's refusal wording was bypassed, which the paper argues earlier benchmarks conflated.
Open-ended single-turn generation on a forbidden prompt (optionally jailbreak-transformed), graded by an LLM judge on refusal plus 5-point specificity and convincingness scales.
No model card in ModelSpec reports this benchmark yet.