{
 "body": "\n## What it measures\n\nToxiGen focuses on a specific failure mode: hate speech that does not use slurs, insults, or other obviously flaggable language, but is still hateful toward a targeted group once read in context. The authors generated the dataset by prompting a large pretrained language model with carefully designed demonstrations and an adversarial classifier-in-the-loop decoding method, producing pairs of toxic and superficially similar benign statements about 13 minority groups. The point is to stress-test toxicity detectors and language models on exactly the cases where surface-level keyword filtering fails.\n\nAs a benchmark, ToxiGen is used two ways: to score a purpose-built toxicity classifier's accuracy at separating its toxic and benign statements, and to see whether a general chat model produces or endorses toxic completions when prompted with ToxiGen-style text.\n\n## How it is scored\n\nThere is no single official ToxiGen leaderboard metric; the paper's own reference use is fine-tuning HateBERT and RoBERTa classifiers on ToxiGen data and reporting their accuracy at detecting toxic versus benign statements, both on ToxiGen's own held-out examples and on other public hate-speech datasets. Human annotators separately confirmed that 94.5% of the machine-generated \"toxic\" statements were indeed judged hate speech, which the authors use as a quality check on the generation pipeline rather than a model score. When ToxiGen appears as a line item in a model or system card, it is typically measuring either a safety classifier's accuracy on ToxiGen examples or a chat model's rate of producing toxic completions when prompted with ToxiGen-style inputs - two different measurements that should not be compared to each other directly.\n\n## Dataset and licence\n\nThe dataset contains 274,000 machine-generated statements, toxic and benign, spanning 13 minority groups, released under a CC BY 4.0 licence. A follow-up release added 27,450 human annotations over a subset of the generated text. The statements and (where released) their human labels are public rather than held out.\n\n## Who publishes it\n\nToxiGen comes from Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray and Ece Kamar, published at ACL 2022 as \"ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection,\" with Microsoft Research among the contributing institutions. The GitHub repository, `microsoft/TOXIGEN`, was archived and made read-only on 2026-07-15; the dataset itself remains accessible through Hugging Face.\n\n## Lineage\n\nToxiGen was built specifically to probe implicit and adversarial toxicity, distinguishing it from earlier hate-speech datasets built from real social-media posts with explicit slurs. Microsoft Research later published a related \"(De)ToxiGen\" research blog post on using the dataset to build more robust detectors; no distinct successor benchmark id for ToxiGen exists elsewhere in this repository's corpus.\n\n## Saturation and contamination\n\nNo maintained third-party leaderboard for ToxiGen was found, so its current ceiling is unknown rather than confirmed saturated or open. Contamination risk sits at medium: the full statement set has been public since 2022 without a described held-out split, and the source repository going read-only in 2026 removes any prospect of a refreshed or rotated test set going forward.\n\n## How to run it\n\nThe reference generation and evaluation code lives in the (now archived) `microsoft/TOXIGEN` GitHub repository, alongside the released HateBERT and RoBERTa classifier checkpoints fine-tuned on ToxiGen data. No lm-evaluation-harness or other standard-harness task name for ToxiGen was confirmed in the sources reviewed for this page.\n\n## Reading the numbers\n\nA high ToxiGen score for a purpose-built classifier means it is good at telling implicitly toxic statements from superficially similar benign ones about the same groups, which is a narrower and harder skill than generic profanity filtering. A low toxic-completion rate for a chat model on ToxiGen prompts is a safety signal, not a completeness one - it says nothing about the model's usefulness or its behavior on toxicity types the dataset does not cover. Because the two use cases produce different kinds of numbers, always check which one a reported ToxiGen score refers to before comparing it across models.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "toxigen",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama Guard 3 8B",
   "model_id": "meta/llama-guard-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 97.5,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama Guard 3 8B INT8",
   "model_id": "meta/llama-guard-3-8b-int8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 97.5,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "shieldgemma 27B",
   "model_id": "google/shieldgemma-27b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 96.8,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "shieldgemma 9B",
   "model_id": "google/shieldgemma-9b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 96.1,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama Guard 3 1B",
   "model_id": "meta/llama-guard-3-1b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 95.8,
   "source": "safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4",
   "model_id": "anthropic/claude-opus-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 4.6",
   "model_id": "anthropic/claude-opus-4-6",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 95.1,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, anthropic-system-card-mythos"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5",
   "model_id": "anthropic/claude-3-5-sonnet-20240620",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3.5 v2",
   "model_id": "anthropic/claude-3-5-sonnet-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4",
   "model_id": "anthropic/claude-sonnet-4-20250514",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5",
   "model_id": "anthropic/claude-sonnet-4-5-20250929",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 4.5 (latest)",
   "model_id": "anthropic/claude-sonnet-4-5",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 94.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Opus 3",
   "model_id": "anthropic/claude-3-opus-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 93.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4",
   "model_id": "openai/gpt-4",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1",
   "model_id": "openai/gpt-4-1",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, domain-evals preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 mini",
   "model_id": "openai/gpt-4-1-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4.1 nano",
   "model_id": "openai/gpt-4-1-nano",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4 Turbo",
   "model_id": "openai/gpt-4-turbo",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 93.2,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Sonnet 3",
   "model_id": "anthropic/claude-3-sonnet-20240229",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o",
   "model_id": "openai/gpt-4o",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-05-13)",
   "model_id": "openai/gpt-4o-2024-05-13",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-08-06)",
   "model_id": "openai/gpt-4o-2024-08-06",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o (2024-11-20)",
   "model_id": "openai/gpt-4o-2024-11-20",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "GPT-4o mini",
   "model_id": "openai/gpt-4o-mini",
   "provider": "openai",
   "provider_display": "OpenAI",
   "score": 92.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5",
   "model_id": "anthropic/claude-3-5-haiku-20241022",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3.5 (latest)",
   "model_id": "anthropic/claude-3-5-haiku-latest",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 92.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash",
   "model_id": "google/gemini-2-0-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.8,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Claude Haiku 3",
   "model_id": "anthropic/claude-3-haiku-20240307",
   "provider": "anthropic",
   "provider_display": "Anthropic",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro",
   "model_id": "google/gemini-2-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals, llm-stats, intlpull"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 05-06",
   "model_id": "google/gemini-2-5-pro-preview-05-06",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview 06-05",
   "model_id": "google/gemini-2-5-pro-preview-06-05",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.5 Pro Preview TTS",
   "model_id": "google/gemini-2-5-pro-preview-tts",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, domain-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Pro",
   "model_id": "google/gemini-1-5-pro",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 91.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, llm-stats, intlpull, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash",
   "model_id": "google/gemini-1-5-flash",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 1.5 Flash-8B",
   "model_id": "google/gemini-1-5-flash-8b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 90.5,
   "source": "lmarena.ai, provider-reports, llm-stats, intlpull, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral Large (latest)",
   "model_id": "mistral/pixtral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 90.5,
   "source": "multimodal-evals, domain-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Maverick 17B 128E Instruct",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 89.8,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Maverick-17B-128E-Instruct-FP8",
   "model_id": "meta/llama-4-maverick-17b-128e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 89.8,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Gemini 2.0 Flash Lite",
   "model_id": "google/gemini-2-0-flash-lite",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 89.5,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E",
   "model_id": "meta/llama-4-scout-17b-16e",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 4 Scout 17B 16E Instruct",
   "model_id": "meta/llama-4-scout-17b-16e-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, llm-stats, domain-evals, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-4-Scout-17B-16E-Instruct-FP8",
   "model_id": "meta/llama-4-scout-17b-16e-instruct-fp8",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.5,
   "source": "lmarena.ai, provider-reports, llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.3 70B Instruct NVFP4",
   "model_id": "nvidia/llama-3-3-70b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 88.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama-3.3-70B-Instruct",
   "model_id": "meta/llama-3-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 88.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5-VL 72B Instruct",
   "model_id": "qwen/qwen2-5-vl-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 87.8,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Command R+",
   "model_id": "cohere/command-r-plus-08-2024",
   "provider": "cohere",
   "provider_display": "Cohere",
   "score": 87.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision",
   "model_id": "meta/llama-3-2-90b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 87.5,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 90B Vision Instruct",
   "model_id": "meta/llama-3-2-90b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 87.5,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Pixtral 12B",
   "model_id": "mistral/pixtral-12b",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 86.8,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B",
   "model_id": "meta/llama-3-1-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 70B Instruct",
   "model_id": "meta/llama-3-1-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 86.5,
   "source": "lmarena.ai, provider-reports, safety-evals, domain-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large (latest)",
   "model_id": "mistral/mistral-large-latest",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 2.1",
   "model_id": "mistral/mistral-large-2411",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals,, domain-evals open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mistral Large 3",
   "model_id": "mistral/mistral-large-2512",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 86.2,
   "source": "lmarena.ai, provider-reports, multimodal-evals, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision",
   "model_id": "meta/llama-3-2-11b-vision",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.8,
   "source": "llm-stats, multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.2 11B Vision Instruct",
   "model_id": "meta/llama-3-2-11b-vision-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.8,
   "source": "multimodal-evals, safety-evals"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2.5 72B Instruct",
   "model_id": "qwen/qwen2-5-72b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 84.8,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek Chat",
   "model_id": "deepseek/deepseek-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2",
   "model_id": "deepseek/deepseek-v2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite",
   "model_id": "deepseek/deepseek-v2-lite",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V2 Lite Chat",
   "model_id": "deepseek/deepseek-v2-lite-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3",
   "model_id": "deepseek/deepseek-v3",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, domain-evals open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3 0324",
   "model_id": "deepseek/deepseek-v3-0324",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.1",
   "model_id": "deepseek/deepseek-v3-1",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2",
   "model_id": "deepseek/deepseek-v3-2",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "DeepSeek V3.2 Exp",
   "model_id": "deepseek/deepseek-v3-2-exp",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 82.5,
   "source": "lmarena.ai, provider-reports, safety-evals, preference-evals, open-llm-leaderboard-v2, llm-stats"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B",
   "model_id": "meta/llama-3-1-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "meta/llama-3-1-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct",
   "model_id": "unsloth/llama-3-1-8b-instruct",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct FP8",
   "model_id": "nvidia/llama-3-1-8b-instruct-fp8",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Llama 3.1 8B Instruct NVFP4",
   "model_id": "nvidia/llama-3-1-8b-instruct-nvfp4",
   "provider": "nvidia",
   "provider_display": "NVIDIA",
   "score": 82.1,
   "source": "lmarena.ai, provider-reports, safety-evals, open-llm-leaderboard-v2"
  }
 ],
 "page": {
  "aliases": [],
  "category": "safety",
  "contamination": {
   "note": "The full 274k-statement dataset has been public on GitHub and Hugging Face since 2022 with no held-out split described in the sources reviewed, so text overlap with later training corpora is plausible. The official GitHub repository was archived (made read-only) on 2026-07-15, though the dataset remains available via Hugging Face.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 274000,
   "size_note": "274,000 machine-generated toxic and benign statements about 13 minority groups; a later release added 27,450 human annotations over a subset.",
   "splits": "not established",
   "url": "https://github.com/microsoft/TOXIGEN"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice H",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "toxigen",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ToxiGen targets implicit and adversarial toxicity - text that is hateful in effect without using slurs or other easily-flagged surface markers - directed at 13 minority groups. It was built by prompting a large language model with a demonstration-based framework and an adversarial classifier-in-the-loop decoding method, producing both toxic and superficially similar benign statements that are hard to tell apart on wording alone. As a model benchmark, it is typically used to test how well a model or a classifier trained on its outputs can separate implicitly toxic text from benign text about the same groups, or how a chat model behaves when asked to complete or discuss ToxiGen-style prompts.",
  "metric": {
   "baseline_note": "The original paper reports human annotators labeled 94.5% of the machine-generated toxic examples as hate speech, and used ToxiGen to fine-tune HateBERT and RoBERTa classifiers, which is the paper's own reference use of the dataset. Exact scoring in a given report depends on whether it measures classifier accuracy on held-out ToxiGen examples or a generative model's toxic-output rate on ToxiGen prompts, which are not the same number.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "classification accuracy / toxicity rate",
   "random_baseline": 50.0,
   "unit": "%"
  },
  "name": "ToxiGen",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2203.09509",
   "title": "ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection",
   "url": "https://arxiv.org/abs/2203.09509",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Thomas Hartvigsen",
    "Saadia Gabriel",
    "Hamid Palangi",
    "Maarten Sap",
    "Dipankar Ray",
    "Ece Kamar"
   ],
   "org": "Microsoft Research",
   "url": "https://github.com/microsoft/TOXIGEN"
  },
  "released": "2022-03",
  "repo_url": "https://github.com/microsoft/TOXIGEN",
  "saturation": {
   "as_of": "",
   "note": "No maintained public leaderboard was found, so a current ceiling could not be established from the sources reviewed; scores in provider system cards are self-reported and not necessarily computed the same way.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (arXiv abstract)",
    "url": "https://arxiv.org/abs/2203.09509"
   },
   {
    "accessed": "2026-09-07",
    "title": "microsoft/TOXIGEN GitHub repository",
    "url": "https://github.com/microsoft/TOXIGEN"
   }
  ],
  "status": "active",
  "subcategory": "implicit hate speech detection",
  "summary": "A machine-generated dataset of 274,000 toxic and benign statements about 13 minority groups, used to test whether models can catch subtle, implicit hate speech.",
  "tags": [
   "safety",
   "toxicity",
   "hate-speech",
   "classifier"
  ],
  "task_format": "Binary classification of statements as toxic or benign (as a classifier benchmark), or completion/discussion of ToxiGen-style prompts scored for toxic output (as a generation safety check)."
 }
}