{
 "body": "\n## What it measures\n\nSWAG tests grounded commonsense inference: given a sentence describing part of a real-world situation, a model must choose which of four candidate sentences most plausibly follows. The situations are drawn from two video-captioning sources -- ActivityNet Captions (short clips of everyday activities) and the Large Scale Movie Description Challenge (audio-description captions from movies) -- so items are physically and socially grounded rather than abstract trivia. One ending is the true next caption; the other three are machine-generated distractors chosen by an adversarial filtering procedure to look plausible without being correct.\n\nThe task is entirely English text, framed as a next-event-prediction problem rather than open-ended generation, so a model only has to rank four given options rather than produce free text.\n\n## How it is scored\n\nEach item is four-way multiple choice, so random guessing scores 25%. The authors built the distractors with an \"Adversarial Filtering\" (AF) procedure: an ensemble of stylistic classifiers is trained to distinguish generated endings from the real one, and endings that fool the ensemble are kept, iteratively raising the difficulty for machine classifiers while (the authors argue) staying easy for humans. To keep the surviving distractors natural, the authors oversampled a large pool of candidate continuations from contemporary language models before filtering. The paper's own validation exercise found human annotators agreed with the gold label about 88% of the time; this page does not repeat that figure as a single authoritative \"human baseline\" since later papers describe it differently, so `metric.human_baseline` is left empty here.\n\n## Dataset and licence\n\nThe dataset totals approximately 113,000 examples, released by Hugging Face's `allenai/swag` \"regular\" configuration as 73,546 train, 20,006 validation, and 20,005 test rows. The official GitHub repository (`rowanz/swagaf`) also ships \"full\" versions of the train and validation files that retain each ending's underlying video source and ordinal human-agreement scores, in addition to the regular 4-choice CSVs used for standard modelling. GitHub's own license detection reports MIT for the repository; the Hugging Face dataset card, by contrast, lists licence as unknown, so this page records the repository's stated licence and flags the disagreement. Test-set labels are withheld in the official release.\n\n## Who publishes it\n\nSWAG is by Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi, at the Paul G. Allen School of Computer Science & Engineering, University of Washington, published as \"SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference\" at EMNLP 2018. The authors continue to host the reference repository and project page at rowanzellers.com/swag.\n\n## Lineage\n\nSWAG has no tracked predecessor in this repository. Its direct successor is HellaSwag (`hellaswag`), built by an overlapping author group after BERT \"soon reached over 86%\" on SWAG (the original BERT paper gives 86.6% dev / 86.3% test for BERT-Large, reached within months of SWAG's release) -- language models had caught up to the adversarial filtering faster than expected, so HellaSwag reran the same idea with a stronger generator and discriminator to reopen a difficulty gap. No other successor or variant id is tracked here.\n\n## Saturation and contamination\n\nSWAG is saturated by its own successor's account: HellaSwag's paper reports that \"BERT soon reached over 86%\" on SWAG (the original BERT paper's own table gives BERT-Large at 86.6% dev / 86.3% test), closing most of the gap to the human-agreement range found during dataset validation and eliminating the separation the adversarial filtering was designed to produce. That is the direct evidence this page uses for `status: saturated`; no current leaderboard or recent top score was found in the sources consulted, since public attention and reporting moved to HellaSwag afterward. Contamination risk is high: train and validation labels have been public and widely mirrored since 2018, and most published SWAG numbers, including lm-evaluation-harness's own, are scored on the validation split rather than the held-out test set.\n\n## How to run it\n\nlm-evaluation-harness implements the task as `swag`, loading `allenai/swag` (config `regular`) with `output_type: multiple_choice`, scoring by both raw accuracy (`acc`) and length-normalised accuracy (`acc_norm`) over the four `ending0`-`ending3` fields, trained on `train` and evaluated on `validation` (there is no configured test split, consistent with the official test labels being withheld). OpenCompass, HELM, and BIG-bench task lists were checked in the sources opened for this page and none showed a SWAG scenario. The authors' own repository provides the adversarial-filtering code and reference data files for anyone re-deriving the dataset.\n\n## Reading the numbers\n\nA high SWAG score today mostly confirms that a model handles physically grounded, video-caption-style commonsense about as well as this particular adversarial-filtering pass could make hard, which by the late 2010s already described strong transformer encoders. Because BERT-Large reached the high 80s not long after release, SWAG stopped separating capable models well before the current generation, and no model card in this repository is expected to lean on it as a meaningful differentiator. Treat SWAG as a historical checkpoint in the commonsense-benchmark lineage rather than a live signal, and prefer its successor HellaSwag, which is still occasionally reported, when comparing current models on the same underlying task design.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swag",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Situations With Adversarial Generations"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Train and validation labels have been public since 2018 and are widely mirrored (GitHub, Hugging Face). Test labels are withheld in the official release, but most published SWAG numbers, including lm-evaluation-harness's, are scored on the validation split, whose labels have been public for years.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT (per the rowanz/swagaf GitHub repository's LICENSE file); the Hugging Face dataset card lists licence as unknown",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 113000,
   "size_note": "The paper and dataset card both state approximately 113k multiple-choice questions in total. The Hugging Face `allenai/swag` \"regular\" configuration splits this as 73,546 train, 20,006 validation, and 20,005 test rows (totalling ~113.5k); the official GitHub repository additionally distributes \"full\" variants of the train/validation files that keep the ordinal annotation scores and video/source metadata behind each of the four endings, not only the regular 4-choice CSVs used for modelling.\n",
   "splits": "train 73,546 / validation 20,006 / test 20,005 (test labels withheld)",
   "url": "https://huggingface.co/datasets/allenai/swag"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-006 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-006"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "swag",
   "opencompass": "",
   "other": ""
  },
  "id": "swag",
  "last_updated": "",
  "leaderboard_url": "https://rowanzellers.com/swag/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [
    "hellaswag"
   ],
   "variants": []
  },
  "measures": "SWAG tests grounded commonsense inference: given a sentence describing part of a real-world situation (drawn from video captioning corpora), a model must pick which of four candidate sentences most plausibly follows. One ending is the true next caption; the other three are adversarially selected machine-generated distractors designed to look plausible on the surface while being wrong. The source situations come from ActivityNet Captions (short YouTube activity clips) and the Large Scale Movie Description Challenge (LSMDC, movie audio-description captions), so the task is English text only, though it is grounded in described physical and social activity rather than abstract text.\n",
  "metric": {
   "baseline_note": "Random guessing on the four-way task scores 25%. The paper's own headline result was that then-state-of-the-art models (ESIM+ELMo) reached about 59% while human annotators who validated the dataset agreed with the gold label roughly 88% of the time in that validation exercise; this page did not independently re-derive a single \"human baseline\" figure comparable across sources, so `human_baseline` is left empty rather than repeating a secondhand number.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "SWAG (Situations With Adversarial Generations)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1808.05326",
   "title": "SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference",
   "url": "https://arxiv.org/abs/1808.05326",
   "year": 2018
  },
  "publisher": {
   "authors": [
    "Rowan Zellers",
    "Yonatan Bisk",
    "Roy Schwartz",
    "Yejin Choi"
   ],
   "org": "Paul G. Allen School of Computer Science & Engineering, University of Washington",
   "url": "https://rowanzellers.com/swag/"
  },
  "released": "2018-08",
  "repo_url": "https://github.com/rowanz/swagaf",
  "saturation": {
   "as_of": "",
   "note": "SWAG was retired by its own authors' next paper: HellaSwag (2019) reports that \"BERT soon reached over 86%, almost human-level performance\" on SWAG. The original BERT paper (Devlin et al. 2018) gives the specific figure behind that claim: BERT-Large at 86.6% dev / 86.3% test on SWAG, reached within months of SWAG's August 2018 release. That closes in on the paper's own ~88% human-agreement figure and eliminates the difficulty gap the adversarial filtering was built to create. That is the basis for treating SWAG as saturated; no independent current leaderboard or recent top score was found in the sources opened for this page, since public benchmarking attention moved to HellaSwag.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference",
    "url": "https://arxiv.org/abs/1808.05326"
   },
   {
    "accessed": "2026-09-08",
    "title": "SWAG (full text, ar5iv)",
    "url": "https://ar5iv.labs.arxiv.org/html/1808.05326"
   },
   {
    "accessed": "2026-09-08",
    "title": "rowanz/swagaf repository",
    "url": "https://github.com/rowanz/swagaf"
   },
   {
    "accessed": "2026-09-08",
    "title": "rowanz/swagaf data directory (train/val/test CSVs, full variants)",
    "url": "https://github.com/rowanz/swagaf/tree/master/data"
   },
   {
    "accessed": "2026-09-08",
    "title": "GitHub API: rowanz/swagaf license detection (MIT)",
    "url": "https://api.github.com/repos/rowanz/swagaf/license"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/swag dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/allenai/swag"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/swag dataset API, Hugging Face",
    "url": "https://huggingface.co/api/datasets/allenai/swag"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness swag task YAML",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/swag/swag.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "HellaSwag: Can a Machine Really Finish Your Sentence? (Zellers et al., arXiv:1905.07830), states BERT 'soon reached over 86%' on SWAG",
    "url": "https://arxiv.org/abs/1905.07830"
   },
   {
    "accessed": "2026-09-08",
    "title": "BERT (Devlin et al., arXiv:1810.04805), Table 4: BERT-Large 86.6% dev / 86.3% test on SWAG",
    "url": "https://ar5iv.labs.arxiv.org/html/1810.04805"
   }
  ],
  "status": "saturated",
  "subcategory": "grounded commonsense inference: predicting the next event in a video-caption situation",
  "summary": "113k four-way multiple-choice questions asking which of four captions plausibly continues a video-derived situation; HellaSwag's direct predecessor.",
  "tags": [
   "commonsense",
   "multiple-choice",
   "adversarial-filtering",
   "nli",
   "saturated"
  ],
  "task_format": "Four-way multiple choice: given a context sentence, select the most plausible of four candidate continuations."
 }
}