{
 "body": "\n## What it measures\n\nSocial IQa (SIQA) tests commonsense reasoning about people's motivations, reactions and mental states in everyday social interactions, rather than reasoning about the physical world the way contemporaries such as PIQA do. Each item gives a short context describing an interaction -- for example, \"Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy\" -- plus a question about intent, effect or reaction, such as \"Why did Jordan do this?\", with three candidate answers. Contexts were seeded from event tuples in the ATOMIC commonsense knowledge graph. The authors designed a specific crowdsourcing framework to reduce a known artifact of earlier multiple-choice datasets: rather than having one worker write both a correct and an incorrect answer to the same question, which tends to give wrong answers a detectable \"wrongness\" in their surface style, incorrect answers were instead sourced as the correct answers to a different, related question, making them harder to rule out on style alone.\n\n## How it is scored\n\nSystems pick one of three answer choices per question, scored by plain accuracy. 33.3% is the random-guess rate for three options. The paper separately reports human accuracy on Dev (86.9%) and Test (84.4%), each measured on a random sample of 900 examples, against contemporary 2019 baselines of GPT at 63.3/63.0%, BERT-base at 63.3/63.1%, and its strongest model, BERT-large, at 66.0/64.5% -- a roughly 21-point gap on Dev, consistent with the paper's own claim of a \">20% gap\" between its best baseline and human performance. An ablation in the paper also found that removing either the context or the question from BERT-large's input collapsed its accuracy toward random, confirming both pieces are actually necessary rather than the question alone leaking the answer.\n\n## Dataset and licence\n\nThe official Hugging Face mirror (`allenai/social_i_qa`) and the `lighteval/siqa` mirror that lm-evaluation-harness actually reads both total 35,364 examples: 33,410 train and 1,954 validation, with no test split hosted in either. The paper's own headline figure is \"38,000\" questions, and its Table 1 reports roughly 33k/2k/2k train/dev/test tuples -- consistent with the public total plus a further ~2,000-question test set whose answers were never included in either public mirror checked for this page. No licence tag or statement was found on the Hugging Face dataset card, and no separate licence file was confirmed in the sources checked for this page.\n\n## Who publishes it\n\nSocial IQa comes from Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras and Yejin Choi, jointly affiliated with the Allen Institute for Artificial Intelligence and the Paul G. Allen School of Computer Science & Engineering at the University of Washington, presented at EMNLP-IJCNLP 2019 (ACL Anthology D19-1454). AI2 historically hosted a submissions leaderboard for the benchmark on its leaderboard.allenai.org site; that subdomain fails to resolve at all today, the same DNS failure found while researching the sibling AI2 benchmark `openbookqa`, and no replacement leaderboard was found.\n\n## Lineage\n\nSocial IQa has no predecessor or successor tracked in this repository. It is frequently bundled alongside contemporaries such as HellaSwag and PIQA (`piqa`) as part of standard commonsense-reasoning eval suites -- this repository's own `commonsense_qa` page notes the same grouping -- without any one of them being a formal successor to another. Within this research batch specifically, it is the social-commonsense counterpart to `openbookqa` and shares a research-community lineage (AI2, and overlapping authorship habits) with `arc_challenge` and `commonsense_qa`, all part of the same 2018-2019 wave of four- or five-way multiple-choice reasoning benchmarks that predate MMLU-era evaluation.\n\n## Saturation and contamination\n\nNo current top score could be confirmed for this page: AI2's leaderboard subdomain fails to resolve at all, and no other maintained leaderboard was found. Combined with a 2019 human ceiling already at 86.9% Dev / 84.4% Test, a best 2019 baseline already around 66%, and zero mentions of \"siqa\" or \"social iqa\" anywhere in this repository's own model-card corpus, the benchmark is treated here as saturated and effectively retired from current frontier-model reporting. Contamination risk is high: the validation split, the only split any harness reviewed for this page actually scores, has carried public answers since 2019, and the underlying ATOMIC knowledge graph its contexts are drawn from has been public even longer.\n\n## How to run it\n\nlm-evaluation-harness implements `social_iqa`, filed under its `siqa` task directory, but reads a third-party Hugging Face mirror (`lighteval/siqa`) rather than the original `allenai/social_i_qa`; the two mirrors report identical split counts, so this is a naming rather than a data difference. It scores multiple-choice log-likelihood accuracy on the validation split only, since no test-split answers are available. OpenCompass ships several `siqa` config variants, some generation-based and some perplexity/log-likelihood-based, rather than a single fixed protocol, so a reported OpenCompass \"siqa\" number should be checked against which config produced it before comparing across papers.\n\n## Reading the numbers\n\nA high Social IQa score mostly confirms a model can track intent and reaction in short, simple social vignettes, a capability that already looked close to solved by 2019-era transformer baselines and has had no maintained leaderboard to track since. Because the only scored split has been public for years and the benchmark is absent from this repository's own current model cards, treat any reported number as a floor check rather than a meaningful differentiator between modern frontier models, and prefer a newer, harder social- or theory-of-mind reasoning benchmark if one is available for the comparison you actually need.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "siqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SocialIQA",
   "Social Interaction QA"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "The validation split, which is the only split any harness reviewed for this page actually scores, has carried public answers since 2019. The paper's crowdsourced contexts draw on the ATOMIC knowledge graph, itself long public, giving a second route by which similar content could reach pretraining data even independent of the exact benchmark items.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Not established: no licence tag or statement was found on the Hugging Face dataset card, and no separate licence file was confirmed in the sources checked for this page.\n",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 35364,
   "size_note": "The official Hugging Face mirror (allenai/social_i_qa) and the lighteval/siqa mirror that lm-evaluation-harness reads both total 35,364 examples: 33,410 train and 1,954 validation, with no test split hosted in either. The paper's own headline figure is \"38,000\" questions, and its Table 1 reports roughly 33k/2k/2k train/dev/test tuples -- consistent with the public total plus a further ~2,000-question test set whose answers were never included in either public mirror checked for this page.\n",
   "splits": "train (33,410) / validation (1,954); the paper's own ~2,000-question test split was never included in the public mirrors checked for this page",
   "url": "https://huggingface.co/datasets/allenai/social_i_qa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 3, slice B",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "social_iqa (lives under the lm_eval/tasks/siqa directory; dataset_path lighteval/siqa, a third-party mirror rather than the original allenai/social_i_qa; output_type multiple_choice; scores accuracy on the validation split)",
   "opencompass": "siqa (OpenCompass ships several generation-based and perplexity/log-likelihood-based config variants rather than a single scoring protocol)",
   "other": ""
  },
  "id": "siqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Social IQa (SIQA) tests commonsense reasoning about people's motivations, reactions and mental states in everyday social interactions, rather than reasoning about the physical world the way contemporaries such as PIQA do. Each item gives a short context describing an interaction -- for example, \"Jordan wanted to tell Tracy a secret, so Jordan leaned towards Tracy\" -- plus a question about intent, effect or reaction, such as \"Why did Jordan do this?\", with three candidate answers. Contexts were seeded from event tuples in the ATOMIC commonsense knowledge graph. The authors designed a specific crowdsourcing framework to reduce a known artifact of earlier multiple-choice datasets: rather than having one worker write both a correct and an incorrect answer to the same question (which tends to give wrong answers a detectable \"wrongness\" in their surface style), incorrect answers were instead sourced as the correct answers to a different, related question, making them harder to rule out on style alone.\n",
  "metric": {
   "baseline_note": "33.3% is the three-way random-guess rate. The paper reports separate human-accuracy figures for Dev (86.9%) and Test (84.4%), each measured on a random sample of 900 examples and marked with an asterisk in its results table; 86.9% is recorded here because Dev/validation is the split every current harness actually scores (see Dataset and licence). Its 2019 baselines on the same Dev/Test splits were GPT at 63.3/63.0%, BERT-base at 63.3/63.1%, and BERT-large at 66.0/64.5% -- consistent with the paper abstract's claim of a \">20% gap\" between its strongest baseline and human performance.\n",
   "direction": "higher_is_better",
   "human_baseline": 86.9,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 33.3,
   "unit": "%"
  },
  "name": "Social IQa",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1904.09728",
   "title": "Social IQa: Commonsense Reasoning about Social Interactions",
   "url": "https://arxiv.org/abs/1904.09728",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Maarten Sap",
    "Hannah Rashkin",
    "Derek Chen",
    "Ronan Le Bras",
    "Yejin Choi"
   ],
   "org": "Allen Institute for Artificial Intelligence (AI2); Paul G. Allen School of Computer Science & Engineering, University of Washington",
   "url": "https://allenai.org"
  },
  "released": "2019-04",
  "repo_url": "",
  "saturation": {
   "as_of": "",
   "note": "No current top score could be confirmed for this page: AI2's leaderboard subdomain (leaderboard.allenai.org), which historically hosted Social IQa submissions, fails to resolve at all (the same DNS failure found while researching `openbookqa`, a sibling AI2 benchmark from the same period), and no other maintained leaderboard was found. Combined with a 2019 human ceiling already at 86.9% Dev / 84.4% Test, a best 2019 baseline already at roughly 66%, and zero mentions of \"siqa\" or \"social iqa\" anywhere in this repository's own model-card corpus (checked by grep), the benchmark is treated here as saturated and effectively retired from current frontier-model reporting rather than actively tracked.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Social IQa: Commonsense Reasoning about Social Interactions (arXiv abstract)",
    "url": "https://arxiv.org/abs/1904.09728"
   },
   {
    "accessed": "2026-09-08",
    "title": "Social IQa (ar5iv full text, incl. Table 2 baseline scores and Section 3.5 data statistics)",
    "url": "https://ar5iv.org/abs/1904.09728"
   },
   {
    "accessed": "2026-09-08",
    "title": "Social IQa: Commonsense Reasoning about Social Interactions, ACL Anthology (EMNLP-IJCNLP 2019)",
    "url": "https://aclanthology.org/D19-1454/"
   },
   {
    "accessed": "2026-09-08",
    "title": "allenai/social_i_qa dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/allenai/social_i_qa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lighteval/siqa dataset (the mirror lm-evaluation-harness's social_iqa task actually reads)",
    "url": "https://huggingface.co/datasets/lighteval/siqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness siqa task config (task name social_iqa)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/tasks/siqa/siqa.yaml"
   }
  ],
  "status": "saturated",
  "subcategory": "social-commonsense multiple-choice question answering",
  "summary": "35,364 public three-way multiple-choice questions about people's motivations and reactions in social situations; a 2019 AI2/UW benchmark with a dead leaderboard and no current model-card coverage.",
  "tags": [
   "multiple-choice",
   "social-commonsense",
   "reasoning",
   "retired-leaderboard"
  ],
  "task_format": "Three-way multiple-choice question answering over a short social context and question, typically zero- or few-shot, English only; no supporting passage beyond the one- or two-sentence context is supplied.\n"
 }
}