{
 "body": "\n## What it measures\n\nWebQuestions measures open-domain question answering. A model receives a natural-language question and must produce a short answer associated with a Freebase entity or relation.\n\n## How it is scored\n\nlm-evaluation-harness treats webqs as a `multiple_choice` task whose \"choices\" are the accepted answer strings for that item (there are no distractor options), scored with `exact_match` (mean aggregation): the model gets credit if it assigns the highest likelihood to one of the correct answers. Answer aliases that are a strict prefix of another accepted answer are collapsed before scoring. No human baseline was established here.\n\n## Dataset and licence\n\nThe Stanford `web_questions` release has a 3,778-example train split and a 2,032-example test split (5,810 total); the harness evaluates the test split. No licence is stated on the dataset page. The public, unheld-out answer set creates high contamination risk.\n\n## Who publishes it\n\nWebQuestions was introduced by Jonathan Berant, Andrew Chou, Roy Frostig and Percy Liang (Stanford, EMNLP 2013) and is integrated by lm-evaluation-harness. No current leaderboard was established.\n\n## Lineage\n\nWebQuestions has related WebQuestionsSP variants. They should be reported separately because answer grounding and parsing differ.\n\n## Saturation and contamination\n\nSaturation is unknown. Public questions and Freebase-linked answers are old, so training overlap is plausible and likely.\n\n## How to run it\n\nRun lm-evaluation-harness task `webqs`. Record harness revision, answer normalization, and whether aliases are accepted.\n\n## Reading the numbers\n\nA strong score indicates retrieval-like factual answering on this question set. It does not establish current factuality, multi-hop robustness, or reliable answers outside Freebase\u2019s coverage. Pair scores with retrieval and calibration checks.\n\n## Notes\n\nShort-answer benchmarks are sensitive to spelling and entity alias policy. These implementation details can change scores substantially even when the underlying model is unchanged.\n\nThe dataset\u2019s historical connection to Freebase also means that current web knowledge is outside scope. A model can answer correctly from memorization without demonstrating retrieval or evidence citation. Reports should state whether aliases, capitalization, punctuation, and multi-answer matching are normalized. Scores are best compared only across the same harness revision and answer-processing rules.\n\nIt is useful to separate questions requiring a single entity from questions accepting aliases or alternate strings. This reveals whether failures arise from retrieval or answer formatting.\n\nThe benchmark also predates many current facts, so freshness should not be inferred from a correct answer. Use contemporary factuality tests alongside it.\n\nThis keeps the interpretation tied to the released dataset.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "webqs",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "Public questions and answers are old and may occur in training data.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 2032,
   "size_note": "HuggingFace stanfordnlp/web_questions has a train split of 3,778 and a test split of 2,032 examples (5,810 total); lm-evaluation-harness evaluates the 2,032-example test split. The harness README's own figure of 6,642 question/answer pairs does not match the dataset's split counts.",
   "splits": "test",
   "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/webqs"
  },
  "freshness": {
   "luna-stream-a-001 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-a-001"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "webqs",
   "opencompass": "",
   "other": ""
  },
  "id": "webqs",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": [
    "web_questions_sp"
   ]
  },
  "measures": "WebQuestions tests open-domain question answering from natural-language questions. The lm-evaluation-harness task identifies the WebQuestions task and its answer-extraction protocol.",
  "metric": {
   "baseline_note": "lm-evaluation-harness scores webqs as a multiple_choice task whose choices are the item's own accepted answers (no distractors); exact_match is 1 if the model assigns highest likelihood to a correct answer.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "exact match",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "WebQuestions",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Semantic Parsing on Freebase from Question-Answer Pairs",
   "url": "https://aclanthology.org/D13-1160/",
   "year": 2013
  },
  "publisher": {
   "authors": [
    "Jonathan Berant",
    "Andrew Chou",
    "Roy Frostig",
    "Percy Liang"
   ],
   "org": "Stanford NLP Group",
   "url": "https://worksheets.codalab.org/worksheets/0xba659fe363cb46e7a505c5b6a774dc8a"
  },
  "released": "2013",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness",
  "saturation": {
   "as_of": "",
   "note": "No current leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness WebQuestions task (README, webqs.yaml, utils.py)",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/webqs"
   },
   {
    "accessed": "2026-09-08",
    "title": "Berant, Chou, Frostig, Liang (2013), Semantic Parsing on Freebase from Question-Answer Pairs",
    "url": "https://aclanthology.org/D13-1160/"
   },
   {
    "accessed": "2026-09-08",
    "title": "stanfordnlp/web_questions dataset API (split sizes)",
    "url": "https://huggingface.co/api/datasets/stanfordnlp/web_questions"
   }
  ],
  "status": "active",
  "subcategory": "question answering",
  "summary": "WebQuestions evaluates short-answer questions whose answers are grounded in Freebase entities.",
  "tags": [
   "question-answering",
   "freebase",
   "open-domain"
  ],
  "task_format": "Natural-language question followed by a short free-form answer."
 }
}