{
 "body": "\n## What it measures\n\nPIQA tests physical commonsense reasoning: given a goal stated in a short sentence, such as how to make a\nhole in a piece of wood, and two candidate solutions, a model must pick the more physically sensible one.\nThe dataset was built from instructables.com, a site of how-to instructions for building, cooking and\neveryday physical tasks, so the knowledge required concerns how everyday materials and actions behave in\nthe physical world rather than facts a model could simply recite. Solutions were engineered to require\nchoosing between a typical, correct-seeming approach and an atypical or physically implausible one, not\nbetween an obviously right and an obviously wrong answer.\n\n## How it is scored\n\nModels choose between exactly two candidate solutions per goal, so random guessing scores 50%.\nlm-evaluation-harness reports both raw accuracy (\"acc\") and length-normalised accuracy (\"acc_norm\",\ncorrecting for a model's tendency to prefer longer or shorter completions regardless of content). Human\nperformance is reported at 94.9%, measured by majority vote among annotators who scored at least 90% on a\nqualification task, which the original paper treats as a soft ceiling. The dataset was filtered with\nAFLite, an adversarial-filtering algorithm that uses an ensemble of lightweight classifiers on\nprecomputed embeddings to remove examples a superficial pattern-matcher could already solve, specifically\nto reduce annotation artifacts.\n\n## Dataset and licence\n\nPIQA holds 16,000 training, 2,000 validation and 3,000 test examples. Test-set labels are not public: the\noriginal project's leaderboard works by emailing predictions to the maintainer for scoring rather than by\npublic release, so most published results \u2014 including lm-evaluation-harness's own task \u2014 evaluate against\nthe 2,000-example validation split rather than the true held-out test set. Licensing is stated two\ndifferent ways in the sources checked for this page: the Hugging Face dataset card (ybisk/piqa) lists the\nlicence as \"unknown,\" while the original project site (yonatanbisk.com/piqa) states the Academic Free\nLicense (\"AFL\") v3.0. Both readings are recorded here rather than picking one. English only, text only.\n\n## Who publishes it\n\nPIQA was introduced by Yonatan Bisk, Rowan Zellers and Ronan Le Bras (University of Washington / Allen\nInstitute for AI), Jianfeng Gao (Microsoft Research AI) and Yejin Choi (University of Washington / Allen\nInstitute for AI), posted to arXiv in November 2019 and published at AAAI 2020. The authors continue to\nhost the dataset and leaderboard at yonatanbisk.com/piqa; the Hugging Face mirror (ybisk/piqa) is the copy\nmost current evaluation harnesses actually load from (directly or via a re-hosted parquet mirror).\n\n## Lineage\n\nPIQA names no formal predecessor. It shares authorship and the AFLite adversarial-filtering methodology\nwith several sibling commonsense-reasoning benchmarks from the same research community, including\nHellaSwag (`hellaswag`) and CommonsenseQA (`commonsense_qa`), both of which have their own pages in this\nrepository. None of the three is a strict predecessor or successor of another \u2014 they read as a family of\nsame-era, same-method commonsense benchmarks covering different domains (physical actions, event\ncontinuation, and general commonsense QA respectively) rather than as a single lineage. PIQA has no\nofficial successor.\n\n## Saturation and contamination\n\nPIQA is effectively saturated for current frontier models: the paper's own 94.9% human baseline is a soft\nceiling that 2019-era models already approached on easier items, and general-purpose LLMs now routinely\nscore in the low-to-mid 90s on the public validation split, leaving little room to separate strong models\nfrom each other. No current top score from a frontier-lab announcement was found in this research \u2014 PIQA\nappears mainly as one line inside broad evaluation-harness sweeps rather than as a headline result \u2014 so no\nspecific current top score is recorded here. Contamination risk is high: the training and validation data\nhave been fully public since 2019 and are effectively guaranteed to appear in large web-scraped\npretraining corpora by now, and the one real protection the design offers \u2014 withheld test labels \u2014 depends\non an email-submission leaderboard that has seen little active use in recent years.\n\n## How to run it\n\nlm-evaluation-harness registers the task as `piqa`, evaluating the validation split and reporting both\n`acc` and `acc_norm`; it loads from a re-hosted parquet mirror (`baber/piqa`) rather than the original\n`ybisk/piqa` repository, likely because the original dataset ships a custom loading script that current\nversions of the `datasets` library will not run without explicitly trusting remote code. inspect_evals\nregisters the same task as `inspect_evals/piqa`. OpenCompass configures it under a `piqa` dataset folder.\nNo BIG-bench or HELM implementation was confirmed in this research.\n\n## Reading the numbers\n\nA high PIQA score today mostly confirms a model has assimilated widely available physical-commonsense\ntext, rather than that it reasons live about physical properties, since the validation split nearly\neveryone reports against has been public for years and the benchmark sits close to its human baseline for\nmost current models. PIQA is more useful now as a floor check for smaller or specialised models, where\nscores still vary meaningfully, than as a way to separate frontier models from each other. Because\ndifferent harnesses may report raw or length-normalised accuracy, and because \"PIQA\" almost always means\nthe public validation split rather than the withheld test set, confirm both before comparing two reported\nnumbers.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "piqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Physical Interaction QA",
   "Physical IQA"
  ],
  "category": "reasoning",
  "contamination": {
   "note": "Training and validation data have been fully public since 2019 and are effectively guaranteed to appear in large web-scraped pretraining corpora by now. The one real protection the design offers \u2014 withheld test-set labels \u2014 depends on an email-submission leaderboard that has seen little active use in recent years, so the validation split nearly everyone actually evaluates against should be assumed in-distribution for most current models' training data.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 21000,
   "size_note": "16,000 training, 2,000 validation and 3,000 test examples, per the dataset card. Official test-set labels are not public: the project's leaderboard works by emailing predictions to the maintainer rather than by public scoring, so most published results, including lm-evaluation-harness's own task, evaluate against the 2,000-example validation split rather than the true held-out test set.\n",
   "splits": "16,000 train / 2,000 validation / 3,000 test; test labels are not public",
   "url": "https://huggingface.co/datasets/ybisk/piqa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice A",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "piqa",
   "lm_eval": "piqa",
   "opencompass": "piqa",
   "other": "lm-evaluation-harness's piqa task loads from a re-hosted parquet mirror (dataset path baber/piqa) rather than the original ybisk/piqa repository, likely because the original dataset ships a custom loading script that current versions of the datasets library will not run without explicitly trusting remote code. It evaluates the validation split (not the withheld test split) and reports both acc and acc_norm (length-normalised accuracy). No BIG-bench or HELM implementation was confirmed.\n"
  },
  "id": "piqa",
  "last_updated": "",
  "leaderboard_url": "https://yonatanbisk.com/piqa/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "PIQA tests physical commonsense reasoning: given a goal stated in a short sentence, such as how to make a hole in a piece of wood, and two candidate solutions, a model must pick the more physically sensible one. The dataset was built from instructables.com, a site of how-to instructions for building, cooking and everyday physical tasks, so the knowledge required concerns how everyday materials and actions behave in the physical world rather than facts a model could simply recite. Solutions were engineered to require choosing between a typical, correct-seeming approach and an atypical or physically implausible one, rather than between an obviously right and an obviously wrong answer.\n",
  "metric": {
   "baseline_note": "94.9% human accuracy is a majority vote among annotators who scored at least 90% on a qualification task, as reported in the original paper; the authors treat it as a soft ceiling rather than a formal maximum.\n",
   "direction": "higher_is_better",
   "human_baseline": 94.9,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 50,
   "unit": "%"
  },
  "name": "PIQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1911.11641",
   "title": "PIQA: Reasoning about Physical Commonsense in Natural Language",
   "url": "https://arxiv.org/abs/1911.11641",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Yonatan Bisk",
    "Rowan Zellers",
    "Ronan Le Bras",
    "Jianfeng Gao",
    "Yejin Choi"
   ],
   "org": "",
   "url": "https://yonatanbisk.com/piqa/"
  },
  "released": "2019-11",
  "repo_url": "",
  "saturation": {
   "as_of": "",
   "note": "No current top score from a frontier-lab announcement was found in this research; PIQA appears mainly as one line in broad evaluation-harness sweeps rather than as a headline result today, so no specific top_score is recorded. It is nonetheless read as saturated: the paper's own 94.9% human baseline is a soft ceiling that general-purpose LLMs now routinely approach on the public validation split, leaving little room to separate strong current models from each other.\n",
   "status": "saturated",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "PIQA: Reasoning about Physical Commonsense in Natural Language",
    "url": "https://arxiv.org/abs/1911.11641"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official PIQA project page (licence, leaderboard submission process, venue)",
    "url": "https://yonatanbisk.com/piqa/"
   },
   {
    "accessed": "2026-09-08",
    "title": "ybisk/piqa dataset card (description, splits, licence: unknown)",
    "url": "https://huggingface.co/datasets/ybisk/piqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness piqa.yaml (dataset_path baber/piqa, validation split, acc/acc_norm)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/piqa/piqa.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals piqa task",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/piqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass piqa dataset config",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/piqa"
   }
  ],
  "status": "active",
  "subcategory": "physical commonsense reasoning",
  "summary": "Binary-choice physical commonsense reasoning built from instructables.com how-to text; a 2019 benchmark now close to its human baseline for most current models.",
  "tags": [
   "commonsense",
   "physical-reasoning",
   "binary-choice",
   "classic-nlp"
  ],
  "task_format": "Given a goal sentence and two candidate solutions, the model selects the more physically appropriate solution; exactly one of the two is correct.\n"
 }
}