{
 "body": "\n## What it measures\n\nSports Understanding gives a model a short statement pairing a real athlete with a sport-specific\naction, for example an athlete described as throwing a touchdown or scoring a goal, and asks whether\nthe statement is plausible or implausible. Answering correctly needs domain knowledge of which sport\na given athlete plays and which actions belong to that sport, so the task exercises a narrow,\nfactual form of commonsense (sports trivia plus action-sport matching) rather than general reasoning.\n\n## How it is scored\n\nThe task is a balanced binary multiple-choice classification, scored with BIG-bench's\nmultiple_choice_grade metric (accuracy on the plausible/implausible choice); chance performance is\n0.5 since the two options are roughly balanced by construction. No human baseline figure was found\nin the sources read for this page.\n\n## Dataset and licence\n\nThe task holds 986 multiple-choice examples, generated combinatorially by pairing athletes from four\nmajor North American sports (the exact sports are not itemised beyond \"four major North American\nsports\" plus soccer) with actions that are either appropriate to that athlete's sport or deliberately\nmismatched, producing plausible and implausible statements respectively. No explicit licence is\nstated for the task's word/name lists in the BIG-bench directory itself.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Ethan Kim as part of the broader, multi-author BIG-bench\ncollaboration coordinated by Google researchers. No separate standalone paper describing this task\nin isolation was found; it is documented in the task's own README and task.json, and separately\ndescribed in the BIG-bench Hard paper as one of the tasks selected for that harder subset.\n\n## Lineage\n\nSports Understanding is one of several hundred independent BIG-bench tasks. It was later selected as\none of the 23 tasks that make up BIG-bench Hard (BBH), a subset chosen because prior language models\nhad not exceeded average human rater performance on it in the original BIG-bench evaluation; BBH\ndoes not have its own page in this repository yet. The task has no predecessor or successor tracked\nhere.\n\n## Saturation and contamination\n\nNo maintained standalone leaderboard for this task was found. Its inclusion in BIG-bench Hard signals\nthat, as of BIG-bench's original evaluation, no model tested had beaten average human performance on\nit, but whether current frontier models have since closed or exceeded that gap is not established\nfrom a source read for this page, so saturation status is marked \"watch\" rather than confirmed\nsaturated or open. Contamination risk is medium: the task has been public since 2021 and is further\namplified by its reuse inside the widely adopted BIG-bench Hard subset; its README carries a canary\nGUID, but that convention only protects against crawlers that honour it.\n\n## How to run it\n\nRun as the `sports_understanding` task in the BIG-bench repository\n(`bigbench/benchmark_tasks/sports_understanding`), or via BIG-bench Hard's\n`sports_understanding.json`. lm-evaluation-harness implements several BBH variants of it under its\n`bbh` task group (zero-shot, few-shot and chain-of-thought forms); the confirmed zero-shot task name\nis `bbh_zeroshot_sports_understanding`, which prompts with a plain \"Q: ... A:\" template and extracts\na yes/plausible or no/implausible answer via regex. Scores from the two source repositories, and\nacross BBH's own variant task names, are not directly comparable without checking shot count and\nprompt format.\n\n## Reading the numbers\n\nA high score shows a model has reliable, largely memorisable sports trivia (which athlete plays\nwhich sport) combined with correct sport-action matching, not deeper reasoning. Because the task\nwas originally chosen for BIG-bench Hard on the grounds that models had not yet beaten human raters\non it, a score well above 0.5 on a modern model is a meaningful signal of progress rather than a\nsaturated, uninformative result, but the exact current gap to human performance and the risk that\nfrontier models have seen the fixed item set during training are both not established here.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "sports_understanding",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "The 986-example task file has been publicly downloadable in the BIG-bench GitHub repository since 2021 and is additionally redistributed as part of BIG-bench Hard, a widely used fine-tuning and evaluation subset, increasing the chance the exact items appear in training corpora relative to a less-reused BIG-bench task. The README does carry a canary GUID intended to let maintainers exclude the task from future training corpora, but that convention only protects against crawlers that honour it, so exposure in general web-scale training data cannot be ruled out.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 986,
   "size_note": "986 multiple-choice queries, confirmed from the task's own README and task.json in the BIG-bench repository, generated combinatorially by pairing athletes from four major North American sports plus soccer with sport-appropriate or deliberately mismatched actions.\n",
   "splits": "Single BIG-bench task file of 986 examples; no separate train/validation/test split is defined by the task itself",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/sports_understanding"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-004 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-004"
  },
  "harness": {
   "bigbench": "sports_understanding",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "bbh_zeroshot_sports_understanding",
   "opencompass": "",
   "other": "Also ships as `sports_understanding.json` among the 23 tasks in BIG-bench Hard (BBH). lm-evaluation-harness carries several BBH variants for it (zero-shot, few-shot, and chain-of-thought versions of each) under its `bbh` task group; the confirmed zero-shot task name is `bbh_zeroshot_sports_understanding`. Scores reported under different variant names are not directly comparable without checking shot count and whether chain-of-thought was used.\n"
  },
  "id": "sports_understanding",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Sports Understanding presents a short statement combining a real athlete's name with a sport-specific action (and sometimes a competition), such as an athlete \"threw a touchdown\" or \"scored a goal,\" and asks the model to classify the statement as plausible or implausible. Getting it right requires knowing which sport a given athlete plays and which actions are appropriate to that sport, i.e. domain-specific commonsense and sports knowledge rather than general reasoning.\n",
  "metric": {
   "baseline_note": "The task is a balanced binary choice, so chance performance is 0.5. No human baseline figure is given in the source read for this page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "Multiple choice grade (accuracy on the plausible/implausible binary choice)",
   "random_baseline": 0.5,
   "unit": "accuracy"
  },
  "name": "Sports Understanding",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "Ethan Kim"
   ],
   "org": "Google (BIG-bench collaboration); task author Ethan Kim",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/sports_understanding"
  },
  "released": "2021",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/sports_understanding",
  "saturation": {
   "as_of": "",
   "note": "No maintained standalone leaderboard for this task was found for this page. It is included as one of the 23 tasks in BIG-bench Hard (BBH), a subset selected because the original BIG-bench paper found no prior language model beat average human rater performance on it, which implies it was not saturated at BIG-bench's original 2022 evaluation; whether current frontier models have since saturated it is not established from a source read for this page.\n",
   "status": "watch",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench sports_understanding task directory (README: task description, 986 examples, multiple_choice_grade metric, author, keywords)",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/sports_understanding"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-Bench-Hard repository bbh/ directory listing confirming sports_understanding.json is one of the 23 BBH tasks",
    "url": "https://github.com/suzgunmirac/BIG-Bench-Hard/tree/main/bbh"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness BBH zero-shot sports_understanding.yaml (task name bbh_zeroshot_sports_understanding, prompt template, answer extraction)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/bbh/zeroshot/sports_understanding.yaml"
   }
  ],
  "status": "active",
  "subcategory": "commonsense plausibility judgement about sports actions and athletes",
  "summary": "A BIG-bench task that asks a model to judge whether a made-up sentence pairing an athlete with a sport-specific action is plausible or implausible.",
  "tags": [
   "commonsense",
   "sports",
   "classification",
   "big-bench",
   "bbh"
  ],
  "task_format": "Zero-shot binary multiple-choice classification: the model picks \"plausible\" or \"implausible\" for each of 986 combinatorially generated statements pairing an athlete with an action drawn from four major North American sports plus soccer, with some pairings sport-appropriate and others deliberately mismatched.\n"
 }
}