{
 "body": "\n## What it measures\n\nSocial Support gives a model a post from an online community and a reply, then asks it to judge\nwhether the reply is supportive, neutral or unsupportive of the poster. The task originates from\nWang and Jurgens' 2018 study of support in online communities, which crowdsourced Likert ratings of\nsupport, agreement, politeness and offensiveness for comment pairs drawn from Reddit, StackExchange\nand Wikipedia. BIG-bench turns a slice of that annotated data into a three-way zero-shot\nclassification task, probing a narrow form of social and emotional understanding rather than\ngeneral sentiment analysis.\n\n## How it is scored\n\nBIG-bench scores the task's three-way multiple-choice answer with multiple_choice_grade. The\nunderlying source paper reports macro-F1 for its own classifiers: a supervised BERT-base model\nreaches 0.54, human annotators reach approximately 0.72, and zero-shot GPT-2-family models tested\nas this BIG-bench task score far lower (GPT-2-XL 0.30, GPT-2 0.06). These two metrics are not\nalways reported for the same models, so a BIG-bench multiple_choice_grade number and the paper's\nmacro-F1 figures should not be read as directly comparable without checking which was used.\n\n## Dataset and licence\n\nThe task file holds 897 multiple-choice examples, each a post/reply pair with a three-way label.\nThe underlying annotations come from Wang and Jurgens (2018), who used crowd workers rating\ncomment-reply pairs on five-point Likert scales for support, agreement, politeness and offensiveness,\nrequiring 70% agreement with gold examples during annotator training and reaching an overall\nKrippendorff's alpha of 0.766. No explicit licence is stated in the BIG-bench task directory itself.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench (a large, multi-author collaborative benchmark suite\ncoordinated by Google researchers) by Zijian Wang and David Jurgens, the same authors as the 2018\nEMNLP source paper that produced the underlying annotated data.\n\n## Lineage\n\nSocial Support is one of several hundred independent BIG-bench tasks; it has no predecessor,\nsuccessor or variant tracked in this repository. It shares its data lineage with Wang & Jurgens'\n2018 support-classification study but is not otherwise part of a named benchmark family.\n\n## Saturation and contamination\n\nNo maintained public leaderboard reports this task in isolation, so its saturation status is not\nestablished from a source read for this page; results exist only inside BIG-bench's own aggregate\ntables and scattered per-model result logs. The task has been publicly downloadable since BIG-bench's\n2021 release, and while its README carries a canary GUID meant to support exclusion from future\ntraining data, that convention only protects against crawlers that honour it, so contamination risk\nis assessed as medium rather than confirmed high or low.\n\n## How to run it\n\nRun as the `social_support` task in the BIG-bench repository (`bigbench/benchmark_tasks/social_support`).\nNo other harness (lm-evaluation-harness, inspect_evals, HELM, OpenCompass) implementation was found\nfor this page.\n\n## Reading the numbers\n\nA high multiple_choice_grade on this task shows a model can distinguish supportive from\nunsupportive replies in short, emotionally loaded online text, a narrow social-cognition skill, not\ngeneral sentiment or empathy. Because the human baseline (~0.72 macro-F1) is itself well short of a\nperfect score, this is a task with real headroom rather than one near a ceiling. With no maintained\nleaderboard and only crawler-dependent canary protection, treat any single reported score\ncautiously and check whether it used multiple_choice_grade or macro-F1 before comparing across\nmodels.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "social_support",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "safety",
  "contamination": {
   "note": "The 897-example task file has been publicly downloadable in the BIG-bench GitHub repository since 2021, and BIG-bench as a whole is a widely used training and evaluation resource, so the exact task items are plausibly present in web-scale training corpora. The README does carry a canary GUID intended to let maintainers exclude the task from future training corpora, but that convention only protects against crawlers that honour it, so exposure in general web-scale training data cannot be ruled out.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 897,
   "size_note": "897 multiple-choice queries, confirmed from the task's own README and task.json in the BIG-bench repository. The underlying annotated corpus comes from Wang & Jurgens' 2018 EMNLP paper, which crowdsourced five-point Likert ratings (support, agreement, politeness, offensiveness) for comment-reply pairs drawn from online communities, with annotators required to reach 70% agreement with gold examples during training and achieving 0.766 Krippendorff's alpha overall.\n",
   "splits": "Single BIG-bench task file of 897 examples; no separate train/validation/test split is defined by the task itself",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/social_support"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-004"
  },
  "harness": {
   "bigbench": "social_support",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "social_support",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Social Support gives a model a post from an online community and a reply comment, and asks the model to judge whether the reply is supportive, neutral or unsupportive of the poster. The annotated corpus behind the task spans Reddit, StackExchange and Wikipedia talk-page interactions, not a single platform. It targets a narrow slice of social and emotional understanding: recognising encouragement, empathy or advice versus dismissiveness or hostility in short, informal, emotionally loaded text, not general sentiment polarity.\n",
  "metric": {
   "baseline_note": "The task's own source paper (Wang & Jurgens 2018) reports a trained BERT-base classifier at 0.54 macro-F1 and human annotators at approximately 0.72 macro-F1 on the underlying support-classification task; zero-shot GPT-2 variants scored far lower (GPT-2-XL 0.30, GPT-2 0.06) when tested as a BIG-bench task, showing a large gap to both the supervised baseline and human agreement. BIG-bench itself reports results via multiple_choice_grade rather than macro-F1 for later models, so the two metrics are not always given for the same models.\n",
   "direction": "higher_is_better",
   "human_baseline": 0.72,
   "max_score": 1.0,
   "name": "Macro-F1 against majority-vote crowd labels (BIG-bench also records multiple_choice_grade)",
   "random_baseline": null,
   "unit": "F1"
  },
  "name": "Social Support",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "It's going to be okay: Measuring access to support in online communities",
   "url": "https://aclanthology.org/D18-1004/",
   "year": 2018
  },
  "publisher": {
   "authors": [
    "Zijian Wang",
    "David Jurgens"
   ],
   "org": "Google (BIG-bench collaboration); task authors Zijian Wang and David Jurgens",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/social_support"
  },
  "released": "2018",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/social_support",
  "saturation": {
   "as_of": "",
   "note": "No public leaderboard tracks this task specifically; it is one of several hundred BIG-bench tasks and results appear only in the BIG-bench paper's aggregate tables and in scattered per-task result files in the repository, not a maintained board. Not established from a source read for this page.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench social_support task directory (README: task description, 897 examples, macro-F1, baselines, authors)",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/social_support"
   },
   {
    "accessed": "2026-09-08",
    "title": "Wang & Jurgens, 'It's going to be okay: Measuring access to support in online communities' (EMNLP 2018) -- source paper for the annotated corpus",
    "url": "https://aclanthology.org/D18-1004/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Project data page for Wang & Jurgens 2018: confirms the crowdsourced annotated corpus spans Reddit, StackExchange and Wikipedia (9,032 instances), not a single platform",
    "url": "http://blablablab.si.umich.edu/projects/support"
   }
  ],
  "status": "active",
  "subcategory": "social and emotional understanding: classify supportiveness of a comment in an online conversation",
  "summary": "A BIG-bench task that asks a model to classify a comment from an online support community as supportive, neutral or unsupportive of the post it replies to.",
  "tags": [
   "social-understanding",
   "classification",
   "safety",
   "big-bench"
  ],
  "task_format": "Zero-shot multiple-choice classification. Each of 897 examples presents a post/reply pair and asks the model to pick one of three labels (supportive, neutral, unsupportive); BIG-bench scores it via the multiple_choice_grade metric over the three answer options.\n"
 }
}