{
 "body": "\n## What it measures\n\nHELM Safety is a Stanford CRFM leaderboard that scores a model's refusal and bias behaviour across five existing safety datasets: BBQ, SimpleSafetyTests, HarmBench, AnthropicRedTeam, and XSTest. Together they probe six risk categories CRFM drew from AI developers' acceptable-use policies: violence, fraud, discrimination, sexual content, harassment, and deception.\n\nA model sees single-turn prompts ranging from overtly unsafe requests, through red-teamed jailbreak attempts meant to bypass guardrails, to sensitive-sounding but benign questions. BBQ instead asks multiple-choice questions testing whether the model leans on a social stereotype when context does not support one. HELM Safety checks whether a response complies with a harmful request, refuses appropriately, or wrongly refuses a benign one.\n\n## How it is scored\n\nBBQ is graded by exact-match accuracy against the unbiased answer. The other four scenarios use an LLM judge: CRFM's two judges are GPT-4o (2024-05-13) and Llama 3.1 405B Instruct Turbo, and each response's harmfulness or helpfulness score is the mean of both judges' ratings. Every scenario score is normalized to 0-1, and the leaderboard's \"Mean score\" for a model is the unweighted average of all five; higher is better, 1.0 is the maximum.\n\nThat single number hides protocol differences. CRFM's write-up documents judges sometimes refusing to grade a harmful transcript rather than score it, at rates that vary sharply by judge -- GPT-4o graded every response in one worked example while Claude 3.5 Sonnet, tried as an alternate judge, failed on nearly 27% of them. A separate table in the same write-up shows automated jailbreak suffixes (the GCG method) cutting HarmBench scores by roughly 26% on average, a comparison that is not folded into the Mean score shown on the leaderboard.\n\n## Dataset and licence\n\nHELM Safety runs five separately published datasets rather than one. Per CRFM's write-up: BBQ (Parrish et al., arXiv 2110.08193) contributes 58,492 bias-probing questions; SimpleSafetyTests (Vidgen et al., arXiv 2311.08370) 100 unsafe prompts; HarmBench (Mazeika et al., arXiv 2402.04249) 321 unsafe prompts; AnthropicRedTeam, drawn from Anthropic's public hh-rlhf red-team transcripts, 38,961 attacks; and XSTest (Rottger et al., arXiv 2308.01263) 450 safe-or-unsafe prompts. All five are fully public prompt sets, browsable per-instance on the leaderboard, with no held-out split described for the composite.\n\nCRFM does not state one unified licence for the combined suite, and each source dataset carries whatever licence its own authors published it under -- not independently verified here, so licence is not established for this page. The evaluation framework itself, stanford-crfm/helm, is published under the Apache 2.0 licence.\n\n## Who publishes it\n\nHELM Safety comes from Stanford's Center for Research on Foundation Models (CRFM). The write-up, \"HELM Safety: Towards Standardized Safety Evaluations of Language Models,\" is by Farzaan Kaiyom, Ahmed Ahmed, Yifan Mai, Kevin Klyman, Rishi Bommasani, and Percy Liang, published as a CRFM blog post in November 2024 rather than as a separate arXiv paper. It sits inside the wider HELM project, which Percy Liang, Rishi Bommasani, Tony Lee and a large multi-author team introduced in \"Holistic Evaluation of Language Models\" (arXiv 2211.09110, November 2022); that original paper defines the HELM framework but does not itself cover Safety, which arrived two years later as a separate initiative under the same project. CRFM maintains the public leaderboard and keeps adding newly released models.\n\n## Lineage\n\nHELM Safety is not a revision of one earlier benchmark; it aggregates five pre-existing, independently published benchmarks -- BBQ, SimpleSafetyTests, HarmBench, AnthropicRedTeam, and XSTest -- none of which has its own page in this repository yet. CRFM says it drew its six-category risk taxonomy from prior work behind AIR-Bench, a sibling CRFM leaderboard that scores models against developers' acceptable-use policies; the two leaderboards initially shared the same 24-model evaluation cohort, but AIR-Bench is a related leaderboard under the same HELM umbrella, not a formal predecessor, and it has no page here either. No successor to HELM Safety has been announced.\n\n## Saturation and contamination\n\nCRFM flagged saturation risk at launch, writing in November 2024 that \"the relatively high scores of the top models indicate potential saturation\" and calling for harder benchmarks to keep separating strong models. Reading the live leaderboard on 2026-09-07, that pattern has continued: the top ten models -- GPT-5 nano, o3, gpt-oss-120b, two Claude 4 Sonnet configurations, GPT-5, GPT-5 mini, Kimi K2 Instruct, Claude 3.5 Sonnet, and o1 -- all sit between 0.974 and 0.986 of a 1.0 maximum, too narrow a band for the Mean score alone to separate leading models.\n\nContamination risk is medium. All five datasets publish their prompts openly; the oldest, BBQ and AnthropicRedTeam, have been public for years, long enough to plausibly reach later training or safety-tuning data. HarmBench and XSTest are newer, from 2023-2024. No source consulted here measures actual training-data overlap, but providers are known to safety-tune against red-team and refusal-style prompts like these, which could raise scores for reasons unrelated to general safety.\n\n## How to run it\n\nThe reference implementation lives in the stanford-crfm/helm GitHub repository (Apache 2.0 licence). Its run configuration, run_entries_safety.conf, names the five scenarios exactly as bbq, harm_bench, simple_safety_tests, anthropic_red_team, and xstest; running all five and averaging their normalized scores reproduces the Mean score column. Matching CRFM's published numbers also requires its two judge models, GPT-4o (2024-05-13) and Llama 3.1 405B Instruct Turbo -- whether CRFM kept these same two judges for every model added to the leaderboard after the original 24 is not established, which matters when comparing older and newer rows.\n\nNo other harness checked for this page -- lm-evaluation-harness, Inspect Evals, OpenCompass, or BIG-bench -- was confirmed to carry a matching five-way \"HELM Safety\" aggregate task; some of the five constituent datasets may appear individually elsewhere under their own names, but that was not independently verified here.\n\n## Reading the numbers\n\nA high HELM Safety score means a model avoided the specific harmful completions and biased BBQ answers these five datasets sample, under two particular LLM judges, on prompts that were not adversarially perturbed. CRFM is explicit this is not a safety certification: even leading models, in its own words, are still very likely to show unsafe behaviour outside what these five benchmarks cover. A model can also score well by refusing broadly rather than by telling harmful and benign requests apart -- XSTest exists to catch exactly that failure, so check its sub-score before trusting a strong overall number. Because top scores cluster tightly, treat small Mean-score gaps between leading models as noise rather than a real ranking, and compare a model's HarmBench score against its HarmBench GCG-T score, where published, to see how much of that safety survives an adversarial prompt.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "helm_safety",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama Guard 3 8B",
   "model_id": "meta/llama-guard-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 95.2,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama Guard 3 8B INT8",
   "model_id": "meta/llama-guard-3-8b-int8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 95.2,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "shieldgemma 27B",
   "model_id": "google/shieldgemma-27b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 94.5,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "shieldgemma 9B",
   "model_id": "google/shieldgemma-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 93.2,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama Guard 3 1B",
   "model_id": "meta/llama-guard-3-1b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 92.8,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 90.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 89.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 89.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 88.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 86.8,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 85.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.2,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision",
   "model_id": "meta/llama-3-2-90b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 84.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 84.5,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 84.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.2,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 83.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 83.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 82.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 82.8,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 82.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision",
   "model_id": "meta/llama-3-2-11b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.5,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision Instruct",
   "model_id": "meta/llama-3-2-11b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 72B Instruct",
   "model_id": "qwen/qwen2-5-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 80.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "meta/llama-3-1-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "meta/llama-3-1-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "unsloth/llama-3-1-8b-instruct",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct FP8",
   "model_id": "nvidia/llama-3-1-8b-instruct-fp8",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct NVFP4",
   "model_id": "nvidia/llama-3-1-8b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 78.5,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2",
   "model_id": "deepseek/deepseek-v2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite",
   "model_id": "deepseek/deepseek-v2-lite",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite Chat",
   "model_id": "deepseek/deepseek-v2-lite-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 78.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  }
 ],
 "page": {
  "aliases": [
   "HELM-Safety"
  ],
  "category": "safety",
  "contamination": {
   "note": "All five source datasets publish their prompts openly; the oldest, BBQ and AnthropicRedTeam, have been public for years, long enough to plausibly reach later training or safety-tuning data. HarmBench and XSTest are newer (2023-2024). No source consulted measures actual training-data overlap, but providers are known to safety-tune against red-team and refusal-style prompts like these.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "Five separate datasets, not one: BBQ 58,492 items, AnthropicRedTeam 38,961 items, XSTest 450 items, HarmBench 321 items, SimpleSafetyTests 100 items, per CRFM's write-up.",
   "splits": "",
   "url": ""
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice G",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "run_entries_safety.conf (scenarios: bbq, harm_bench, simple_safety_tests, anthropic_red_team, xstest)",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "helm_safety",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/safety/latest/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "HELM Safety is a Stanford CRFM leaderboard that scores a model's refusal and bias behaviour across five existing safety datasets: BBQ, SimpleSafetyTests, HarmBench, AnthropicRedTeam, and XSTest. Together they probe six risk categories CRFM drew from AI developers' acceptable-use policies: violence, fraud, discrimination, sexual content, harassment, and deception. A model sees single-turn prompts ranging from overtly unsafe requests, through red-teamed jailbreak attempts meant to bypass guardrails, to sensitive-sounding but benign questions; BBQ instead asks multiple-choice questions testing whether the model leans on a social stereotype when context does not support one.\n",
  "metric": {
   "baseline_note": "CRFM's write-up and leaderboard do not publish a random or human baseline for the aggregate Mean score; models are compared to each other, not to a fixed reference point.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Mean score (unweighted average of five normalized per-scenario scores)",
   "random_baseline": null,
   "unit": "0-1 scale"
  },
  "name": "HELM Safety",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "HELM Safety: Towards Standardized Safety Evaluations of Language Models",
   "url": "https://crfm.stanford.edu/2024/11/08/helm-safety.html",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Farzaan Kaiyom",
    "Ahmed Ahmed",
    "Yifan Mai",
    "Kevin Klyman",
    "Rishi Bommasani",
    "Percy Liang"
   ],
   "org": "Stanford Center for Research on Foundation Models (CRFM)",
   "url": "https://crfm.stanford.edu/helm/safety/latest/"
  },
  "released": "2024-11",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "2026-09",
   "note": "CRFM flagged 'potential saturation' among top models at the November 2024 launch. On the live leaderboard read 2026-09-07, the top ten models span only 0.974-0.986 of a 1.0 maximum, confirming that pattern has continued.",
   "status": "saturated",
   "top_score": 0.986
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "HELM Safety leaderboard",
    "url": "https://crfm.stanford.edu/helm/safety/latest/"
   },
   {
    "accessed": "2026-09-07",
    "title": "HELM Safety: Towards Standardized Safety Evaluations of Language Models (CRFM blog)",
    "url": "https://crfm.stanford.edu/2024/11/08/helm-safety.html"
   },
   {
    "accessed": "2026-09-07",
    "title": "Holistic Evaluation of Language Models (HELM) -- leaderboard index",
    "url": "https://crfm.stanford.edu/helm/"
   },
   {
    "accessed": "2026-09-07",
    "title": "Holistic Evaluation of Language Models",
    "url": "https://arxiv.org/abs/2211.09110"
   },
   {
    "accessed": "2026-09-07",
    "title": "stanford-crfm/helm GitHub repository",
    "url": "https://github.com/stanford-crfm/helm"
   },
   {
    "accessed": "2026-09-07",
    "title": "HELM Safety run_entries configuration",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/presentation/run_entries_safety.conf"
   }
  ],
  "status": "active",
  "subcategory": "",
  "summary": "Stanford CRFM leaderboard that scores a model by averaging its results across five existing safety benchmarks into one 0-1 number.",
  "tags": [
   "safety",
   "llm-judge",
   "jailbreak",
   "red-teaming",
   "refusal",
   "bias",
   "aggregate-benchmark",
   "crfm"
  ],
  "task_format": "Single-turn prompts; graded by exact-match accuracy (BBQ) or a two-model LLM-judge harmfulness/helpfulness rating (the other four scenarios)."
 }
}