{
 "body": "\n## What it measures\n\nNewsQA measures reading comprehension over CNN news articles. Given a passage and question, a model must select the answer span supported by that passage. The scenario also retains questions whose consensus says that no answer appears in the passage.\n\nThe task is English text question answering. HELM formats each example as `Passage`, `Question`, and `Answer`, while the dataset was created from articles and crowd-authored questions.\n\n## How it is scored\n\nHELM retains consensus and crowd-worker answer strings as references, including unanswerable cases, and its generic generation metrics can score exact match against accepted references. Reporters should state the HELM metric configuration and treatment of \u201cNo Answer\u201d; the scenario itself does not define token-level normalization beyond those references.\n\n## Dataset and licence\n\nHELM describes 12,744 stories and more than 119,633 question-answer pairs: 92,549 training, 5,166 development, and 5,126 test pairs. It says the questions and answers were written by crowd workers. The original training data cannot be redistributed directly because of copyright restrictions; the HELM scenario requires obtaining it through the original instructions. A dataset licence was not established.\n\n## Who publishes it\n\nThe benchmark is associated with Maluuba Research and the paper \u201cNewsQA: A Machine Comprehension Dataset\u201d (arXiv:1611.09830). The original repository documents data preparation. HELM maintains a runnable scenario, but no current standalone leaderboard was established.\n\n## Lineage\n\nNewsQA is a standalone reading-comprehension dataset. The sources read do not establish a predecessor, successor, or repository variant.\n\n## Saturation and contamination\n\nSaturation is not established. The data and task are public and old, so exposure in training corpora is plausible; the sources read do not quantify that exposure. Article copyright restrictions affect redistribution, but do not by themselves make the evaluation contamination-safe.\n\n## How to run it\n\nHELM\u2019s `newsqa` scenario reads the restricted `combined-newsqa-data-v1.json`, removes questions marked bad, samples one question per article with `random.seed(0)`, and uses train and validation splits. It accepts consensus and crowd-worker answers, including \u201cNo Answer\u201d. Prompt construction, reference selection, and sampling choices can change results.\n\n## Reading the numbers\n\nA strong score indicates reliable extraction from the supplied article and handling of missing answers. It does not measure broad factual knowledge without context, current news knowledge, or long-form explanation. Compare the split, answer normalization, and treatment of unanswerable questions before comparing scores. Report the prompt and whether the restricted data were processed exactly as HELM specifies.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "newsqa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "Articles and questions are publicly documented, but the sources read do not establish training-set exposure for current models.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": 12744,
   "size_note": "12,744 stories and over 119,633 question-answer pairs; the source gives split counts of 92,549 train, 5,166 development, and 5,126 test pairs.",
   "splits": "train, development, test",
   "url": "https://github.com/Maluuba/newsqa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-batch-017 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review, luna-batch-017"
  },
  "harness": {
   "bigbench": "",
   "helm": "newsqa",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "newsqa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "NewsQA presents a CNN article and a natural-language question written by a crowd worker. The answer is normally a span in the article, but some questions have no answer in context.",
  "metric": {
   "baseline_note": "Official human baseline was not established from the sources read.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact match over accepted reference strings (HELM)",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "NewsQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "1611.09830",
   "title": "NewsQA: A Machine Comprehension Dataset",
   "url": "https://arxiv.org/abs/1611.09830",
   "year": 2016
  },
  "publisher": {
   "authors": [
    "Akhilesh N. Rao",
    "et al."
   ],
   "org": "Maluuba Research",
   "url": "https://github.com/Maluuba/newsqa"
  },
  "released": "2016",
  "repo_url": "https://github.com/Maluuba/newsqa",
  "saturation": {
   "as_of": "",
   "note": "No current authoritative leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM NewsQA scenario",
    "url": "https://github.com/stanford-crfm/helm/blob/main/src/helm/benchmark/scenarios/newsqa_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "NewsQA paper",
    "url": "https://arxiv.org/abs/1611.09830"
   },
   {
    "accessed": "2026-09-08",
    "title": "Maluuba NewsQA repository",
    "url": "https://github.com/Maluuba/newsqa"
   }
  ],
  "status": "active",
  "subcategory": "extractive question answering",
  "summary": "NewsQA tests whether a model can answer questions with text spans from CNN news articles, including unanswerable questions.",
  "tags": [
   "question-answering",
   "reading-comprehension",
   "extractive"
  ],
  "task_format": "Passage, question, and answer prompt; extract a span or return No Answer."
 }
}