HarmBench behaviors attacked with GCG suffixes optimized once against four open models and transferred unchanged to the target; a lower Attack Success Rate is the safety-desirable outcome.
unassessed
| Category | safety |
|---|---|
| Subcategory | HarmBench behaviors attacked with precomputed, transferred Greedy Coordinate Gradient (GCG) adversarial suffixes |
| Page status | active |
| Metric | Attack Success Rate (ASR) under transferred GCG suffixes |
| Direction | lower_is_better |
| Unit | % |
| Dataset size | 401 |
| Dataset licence | Not stated. The GitHub repository hosting the precomputed suffixes HELM reads (farzaank/harmbench-gcg-ensembled) has no LICENSE file and essentially no README beyond its title, confirmed by opening both directly; the underlying HarmBench behavior strings themselves are MIT-licensed via the parent centerforaisafety/HarmBench project. |
| Publisher | The GCG-Transfer attack method and its evaluation within HarmBench are published by the Center for AI Safety (CAIS) and co-authors (see harm_bench.md); the specific precomputed suffix file HELM reads is hosted in a separate, minimally documented GitHub repository under the account "farzaank," whose institutional affiliation, if any, was not stated anywhere this page checked. |
harm_bench_gcg_transfer evaluates the same textual behaviors as [HarmBench](harm_bench.md), but instead of the plain, undisguised request, each behavior is appended with an adversarial suffix produced by GCG-Transfer (GCG-T). HarmBench's own paper describes GCG-T as extending GCG-Multi by "simultaneously optimizing against multiple training models" -- specifically Llama 2 7B Chat, Llama 2 13B Chat, Vicuna 7B and Vicuna 13B -- "to yield test cases that can be transferred to all models." Unlike a directly optimized GCG attack, the attacker never touches the model actually being evaluated: a fixed set of suffixes, computed once against those four training models, is applied unchanged to whichever target model is under test. This is a black-box, universal-suffix threat model, meaningfully weaker in general than a per-target, white-box optimized attack, and HarmBench's own results bear that out directly (see Saturation and contamination).
Each HarmBench behavior string is paired with one precomputed GCG-T suffix; the concatenated (behavior + suffix) text is sent to the target model as a single-turn prompt with no further attack logic applied at evaluation time, the model's completion is generated, and a classifier judges whether the completion exhibits the harmful behavior -- structurally identical to HarmBench's own pipeline, just with one fixed, precomputed attack applied to every prompt instead of the unmodified behavior text.
No model card in ModelSpec reports this benchmark yet.