HELM Safety

Stanford CRFM leaderboard that scores a model by averaging its results across five existing safety benchmarks into one 0-1 number.

Also known as: HELM-Safety

unassessed

What it measures

HELM Safety is a Stanford CRFM leaderboard that scores a model's refusal and bias behaviour across five existing safety datasets: BBQ, SimpleSafetyTests, HarmBench, AnthropicRedTeam, and XSTest. Together they probe six risk categories CRFM drew from AI developers' acceptable-use policies: violence, fraud, discrimination, sexual content, harassment, and deception. A model sees single-turn prompts ranging from overtly unsafe requests, through red-teamed jailbreak attempts meant to bypass guardrails, to sensitive-sounding but benign questions; BBQ instead asks multiple-choice questions testing whether the model leans on a social stereotype when context does not support one.

Task format

Single-turn prompts; graded by exact-match accuracy (BBQ) or a two-model LLM-judge harmfulness/helpfulness rating (the other four scenarios).

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub