{
 "body": "\n## What it measures\n\nHellaSwag tests commonsense inference: given a short description of an everyday situation, a model must pick which of four possible continuations is the most plausible next step. Contexts come from two sources -- ActivityNet video captions and WikiHow how-to articles -- covering ordinary physical activities like cooking, repairs or sports. The task looks trivial to a person but was built specifically to be hard for the language models of its time, isolating whether a model has a working sense of how everyday situations unfold rather than just fluent language generation.\n\n## How it is scored\n\nEach item is four-way multiple choice: one human-written correct ending and three machine-generated wrong ones, so random guessing scores 25%. The authors reported human accuracy at 95.6% on the released test set (95.6% in-domain, 95.7% zero-shot), against well under 50% for the strongest models available in 2019. Most published results use the validation split, since the official test-set labels are held out and were originally scored only through the authors' own leaderboard.\n\n## Dataset and licence\n\nThe released dataset totals 59,950 examples: 39,905 for training (14,740 from ActivityNet, 25,165 from WikiHow), 10,042 for validation and 10,003 for test, with each split further divided into \"in-domain\" (activity categories seen in training) and \"zero-shot\" (held-out categories) halves. The paper's own methodology narrative separately describes a larger candidate pool of roughly 70,000 problems (about 25,000 ActivityNet-derived, 45,000 WikiHow-derived) before adversarial filtering and deduplication produced the released splits. The GitHub repository states an MIT licence; validation-set labels are public, test-set labels are withheld.\n\n## Who publishes it\n\nHellaSwag comes from Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi and Yejin Choi at the Paul G. Allen School of Computer Science & Engineering, University of Washington, published as \"HellaSwag: Can a Machine Really Finish Your Sentence?\" at ACL 2019. The authors continue to host the reference repository and a submission leaderboard at rowanzellers.com/hellaswag.\n\n## Lineage\n\nHellaSwag is the direct successor to SWAG (Situations With Adversarial Generations), an earlier commonsense-inference dataset by an overlapping author group that language models caught up to faster than the authors expected. HellaSwag reran the same adversarial-filtering idea with a stronger generator and discriminator to restore a difficulty gap. SWAG does not have a page in this repository, and no successor or variant id is tracked here either.\n\n## Saturation and contamination\n\nThe dataset's own authors closed their public leaderboard to new submissions as of November 2024, and their reported human accuracy of 95.6% leaves little headroom above where current frontier models are commonly described as sitting; both point to a benchmark that no longer separates leading models. No source consulted here gives a specific recent top score. Contamination is plausible from two directions: most published scores use the validation split, whose labels have been public since 2019, and even where true test labels are withheld, the underlying ActivityNet captions and WikiHow articles are separately available across the open web, so a model could learn to favor plausible continuations without ever seeing HellaSwag's own label file. No dedicated HellaSwag contamination study was found in the sources consulted, so this assessment rests on the dataset's public/held-out mechanics rather than a measured leakage rate.\n\n## How to run it\n\nlm-evaluation-harness implements it as the `hellaswag` task, scoring by length-normalized log-likelihood over the four candidate endings rather than free generation, which is the most common way it is reported. The authors' own repository at github.com/rowanz/hellaswag provides the adversarial-filtering code and the reference data files. Because scoring can be done either by likelihood ranking or by prompting a model to output a choice letter, and because reporters mix validation-set and leaderboard test-set numbers, comparing HellaSwag scores across papers is only safe when the scoring method and split match.\n\n## Reading the numbers\n\nA HellaSwag score close to the mid-90s tells you a model handles ordinary, physically grounded commonsense about as well as this dataset can distinguish, which by 2026 describes most current models, so it mainly separates weak or very small models from everything else. A low score is still a real signal of trouble with grounded situational reasoning. Because most reporters score the validation split rather than the held-out test set, and because likelihood-based and generation-based scoring can disagree, treat small differences between reported HellaSwag numbers as noise rather than a genuine capability gap.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "hellaswag",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x22B Instruct v0.1",
   "model_id": "mistral/mixtral-8x22b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 89.1,
   "source": "open-llm-leaderboard-v2, llm-stats, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B Instruct v0.1",
   "model_id": "mistral/mixtral-8x7b-instruct-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 87.6,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Mixtral 8x7B DPO",
   "model_id": "nous-research/nous-hermes-2-mixtral-8x7b-dpo",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 87.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Mixtral 8x7B v0.1",
   "model_id": "mistral/mixtral-8x7b-v0-1",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 86.5,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B",
   "model_id": "01-ai/yi-1-5-34b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 86.1,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat",
   "model_id": "01-ai/yi-1-5-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 86.0,
   "source": "open-llm-leaderboard-v2, open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B Chat 16K",
   "model_id": "01-ai/yi-1-5-34b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 86.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B",
   "model_id": "meta/meta-llama-3-70b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "meta/meta-llama-3-70b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 85.7,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2026-04",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 70B Instruct",
   "model_id": "nous-research/meta-llama-3-70b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 85.7,
   "source": "llm-stats, open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 Yi 34B",
   "model_id": "nous-research/nous-hermes-2-yi-34b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 85.5,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 34B 32K",
   "model_id": "01-ai/yi-1-5-34b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 85.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "falcon 40B",
   "model_id": "tii/falcon-40b",
   "provider": "tii",
   "provider_display": "TII",
   "score": 85.3,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B Instruct v0.2",
   "model_id": "mistral/mistral-7b-instruct-v0-2",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 84.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Nous Hermes 2 SOLAR 10.7B",
   "model_id": "nous-research/nous-hermes-2-solar-10-7b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 84.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B Chat",
   "model_id": "01-ai/yi-34b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 84.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Pro Llama 3 8B",
   "model_id": "nous-research/hermes-2-pro-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 83.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Mistral 7B v0.3",
   "model_id": "mistral/mistral-7b-v0-3",
   "provider": "mistral",
   "provider_display": "Mistral AI",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "mistral 7B v0.3 bnb 4bit",
   "model_id": "unsloth/mistral-7b-v0-3-bnb-4bit",
   "provider": "unsloth",
   "provider_display": "Unsloth",
   "score": 83.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Hermes 2 Theta Llama 3 8B",
   "model_id": "nous-research/hermes-2-theta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 82.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 7B it",
   "model_id": "google/gemma-7b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 82.5,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "meta/meta-llama-3-8b",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 82.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B",
   "model_id": "nous-research/meta-llama-3-8b",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 82.2,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 34B 200K",
   "model_id": "01-ai/yi-34b-200k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 82.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat",
   "model_id": "01-ai/yi-1-5-9b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 80.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B Chat 16K",
   "model_id": "01-ai/yi-1-5-9b-chat-16k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 80.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 4K instruct",
   "model_id": "microsoft/phi-3-mini-4k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 80.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B",
   "model_id": "01-ai/yi-1-5-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 80.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Phi 3 mini 128K instruct",
   "model_id": "microsoft/phi-3-mini-128k-instruct",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 80.1,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 9B 32K",
   "model_id": "01-ai/yi-1-5-9b-32k",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 79.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B base",
   "model_id": "deepseek/deepseek-llm-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 79.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek llm 7B chat",
   "model_id": "deepseek/deepseek-llm-7b-chat",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 79.4,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B Chat",
   "model_id": "01-ai/yi-1-5-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 78.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 9B",
   "model_id": "01-ai/yi-9b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 78.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "meta/meta-llama-3-8b-instruct",
   "provider": "meta",
   "provider_display": "Meta",
   "score": 78.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Meta Llama 3 8B Instruct",
   "model_id": "nous-research/meta-llama-3-8b-instruct",
   "provider": "nous-research",
   "provider_display": "Nous Research",
   "score": 78.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 1.5 6B",
   "model_id": "01-ai/yi-1-5-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 78.0,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B",
   "model_id": "01-ai/yi-6b",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 76.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Yi 6B Chat",
   "model_id": "01-ai/yi-6b-chat",
   "provider": "01-ai",
   "provider_display": "01.AI",
   "score": 76.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "phi 2",
   "model_id": "microsoft/phi-2",
   "provider": "microsoft",
   "provider_display": "Microsoft",
   "score": 74.9,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B",
   "model_id": "google/gemma-2b",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 71.8,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 1.5B Instruct",
   "model_id": "qwen/qwen2-1-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 66.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "OLMo 1B hf",
   "model_id": "allen-ai/olmo-1b-hf",
   "provider": "allen-ai",
   "provider_display": "Allen AI",
   "score": 63.6,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "gemma 2B it",
   "model_id": "google/gemma-2b-it",
   "provider": "google",
   "provider_display": "Google DeepMind",
   "score": 62.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "chatglm2 6B",
   "model_id": "zhipu/chatglm2-6b",
   "provider": "zhipu",
   "provider_display": "Zhipu AI",
   "score": 59.0,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B instruct",
   "model_id": "deepseek/deepseek-coder-6-7b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 55.1,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 6.7B base",
   "model_id": "deepseek/deepseek-coder-6-7b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 53.5,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "Qwen2 0.5B Instruct",
   "model_id": "qwen/qwen2-0-5b-instruct",
   "provider": "qwen",
   "provider_display": "Alibaba / Qwen Team",
   "score": 48.7,
   "source": "open-llm-leaderboard-v1, open-llm-leaderboard-v2"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B base",
   "model_id": "deepseek/deepseek-coder-1-3b-base",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 39.9,
   "source": "open-llm-leaderboard-v1"
  },
  {
   "as_of": "2024-07",
   "attribution": "unverified-legacy",
   "display_name": "deepseek coder 1.3B instruct",
   "model_id": "deepseek/deepseek-coder-1-3b-instruct",
   "provider": "deepseek",
   "provider_display": "DeepSeek",
   "score": 39.9,
   "source": "open-llm-leaderboard-v1"
  }
 ],
 "page": {
  "aliases": [],
  "category": "reasoning",
  "contamination": {
   "note": "Most published scores use the validation split, whose labels have been public since 2019. Even where true test-set labels are withheld, the underlying ActivityNet captions and WikiHow articles are separately available across the open web, so memorization is plausible either way. No dedicated HellaSwag contamination study was found in the sources consulted; this assessment rests on the dataset's public/held-out mechanics rather than a measured leakage rate.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 59950,
   "size_note": "39,905 train (14,740 ActivityNet-sourced, 25,165 WikiHow-sourced) + 10,042 validation + 10,003 test; each split further divides into in-domain and zero-shot halves. The paper's own methodology narrative describes a larger pre-filtering candidate pool of roughly 70,000 problems (about 25,000 ActivityNet, 45,000 WikiHow) before adversarial filtering and deduplication produced the released splits.",
   "splits": "train (39,905) / validation (10,042) / test (10,003)",
   "url": "https://github.com/rowanz/hellaswag/tree/master/data"
  },
  "freshness": {
   "researched": "2026-09-07",
   "researched_by": "sonnet-5 agent, batch 1, slice F",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "hellaswag",
   "opencompass": "",
   "other": ""
  },
  "id": "hellaswag",
  "last_updated": "",
  "leaderboard_url": "https://rowanzellers.com/hellaswag/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "HellaSwag tests commonsense inference: given a short description of an everyday situation, a model must pick which of four possible continuations is the most plausible next step. Contexts are drawn from ActivityNet video captions and WikiHow how-to articles, covering ordinary physical activities such as cooking, repairs or sports. The task looks trivial to a person but was built specifically to be hard for the language models of its time, isolating whether a model has a working sense of how everyday situations unfold rather than just fluent language generation.",
  "metric": {
   "baseline_note": "Random guessing scores 25% on this four-way multiple-choice task; the paper measured human accuracy at 95.6% overall (95.6% in-domain, 95.7% zero-shot).",
   "direction": "higher_is_better",
   "human_baseline": 95.6,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "HellaSwag",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1905.07830",
   "title": "HellaSwag: Can a Machine Really Finish Your Sentence?",
   "url": "https://arxiv.org/abs/1905.07830",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Rowan Zellers",
    "Ari Holtzman",
    "Yonatan Bisk",
    "Ali Farhadi",
    "Yejin Choi"
   ],
   "org": "Paul G. Allen School of Computer Science & Engineering, University of Washington",
   "url": "https://rowanzellers.com/hellaswag/"
  },
  "released": "2019-05",
  "repo_url": "https://github.com/rowanz/hellaswag",
  "saturation": {
   "as_of": "",
   "note": "The dataset's own authors closed their public leaderboard to new submissions as of November 2024, and reported human accuracy is 95.6% overall -- both signal the benchmark's practical retirement for separating current frontier models, though no source consulted here gives a specific recent top score.",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-07",
    "title": "HellaSwag: Can a Machine Really Finish Your Sentence?",
    "url": "https://arxiv.org/abs/1905.07830"
   },
   {
    "accessed": "2026-09-07",
    "title": "HellaSwag (full text, ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1905.07830"
   },
   {
    "accessed": "2026-09-07",
    "title": "HellaSwag - ACL Anthology",
    "url": "https://aclanthology.org/P19-1472/"
   },
   {
    "accessed": "2026-09-07",
    "title": "rowanz/hellaswag repository",
    "url": "https://github.com/rowanz/hellaswag"
   },
   {
    "accessed": "2026-09-07",
    "title": "HellaSwag leaderboard",
    "url": "https://rowanzellers.com/hellaswag/"
   },
   {
    "accessed": "2026-09-07",
    "title": "Rowan/hellaswag dataset card",
    "url": "https://huggingface.co/datasets/Rowan/hellaswag"
   },
   {
    "accessed": "2026-09-07",
    "title": "hellaswag | TensorFlow Datasets catalog",
    "url": "https://www.tensorflow.org/datasets/catalog/hellaswag"
   },
   {
    "accessed": "2026-09-07",
    "title": "EleutherAI lm-evaluation-harness (hellaswag task)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness"
   }
  ],
  "status": "saturated",
  "subcategory": "commonsense natural language inference",
  "summary": "Four-way multiple-choice test of commonsense sentence continuation, built by adversarial filtering to defeat models of its era.",
  "tags": [
   "commonsense",
   "multiple-choice",
   "sentence-completion",
   "adversarial-filtering",
   "nli"
  ],
  "task_format": "Four-way multiple choice; given a short context, the model selects the most plausible of four candidate continuations."
 }
}