{
 "body": "\n## What it measures\n\n`swde` in lm-evaluation-harness gives a model the full text of a webpage \u2014 in this implementation,\npages from the movie vertical of the SWDE structured-web-data corpus \u2014 plus the name of an\nattribute such as \"release date,\" \"genre,\" \"MPAA rating\" or \"director.\" The model must generate the\nvalue of that attribute as it appears on the page. Because the answer is present verbatim somewhere\nin a long, HTML-derived block of text rather than in the model's parametric knowledge, the task is\nbest read as an associative-recall / in-context information-extraction probe: can the model find\nand reproduce a specific fact buried in a longer document, given only a short key describing what\nto look for.\n\nThe task is explicitly a simplified, zero-shot adaptation for smaller language models rather than a\nreimplementation of the full original SWDE wrapper-induction benchmark, which asked systems to\ninduce extraction rules generalizing across many pages of a site.\n\n## How it is scored\n\nScoring uses lm-evaluation-harness's own `contains` metric: `task.py`'s `contains_score` does a\ncase-insensitive regex search for whether the gold value(s) appear anywhere in the model's\ngenerated continuation, taking the best match if multiple valid labels are given, and reports the\nmean over the evaluation set. There is no fixed maximum other than 100% (all items matched) and no\npublished random or human baseline. The task's README flags a protocol change: version 1\n(2026-06-22) strips whitespace from both prompts and targets before matching, which changes scores\nenough that they should not be compared against version 0 runs.\n\n## Dataset and licence\n\nThe harness loads `hazyresearch/based-swde-v2` from Hugging Face, which the dataset card lists as\n1,111 rows drawn from 141 source documents, in a single \"validation\" split (roughly 1.83 MB as\nparquet). Each row carries a document id, filename, attribute key, gold value and the page's full\ntext. No explicit licence is stated on the Hugging Face dataset card for this particular derivative;\nthe underlying page content originates from a real commercial movie-guide website (AMC), and the\noriginal SWDE corpus it descends from was distributed for academic research use. Both the page text\nand the gold answers are visible in the public dataset files.\n\n## Who publishes it\n\nThe dataset and this evaluation format come from the EVAPORATE paper, \"Language Models Enable\nSimple Systems for Generating Structured Views of Heterogeneous Data Lakes\" (Arora, Yang, Eyuboglu,\nNarayan, Hojel, Trummer and R\u00e9, arXiv:2304.09433, 2023), from Stanford's HazyResearch group. The\ngroup also maintains a fork of lm-evaluation-harness (\"based-evaluation-harness\") that packages\n`swde` alongside other recall-intensive tasks for evaluating sub-quadratic and small language\nmodels; the task has since been folded into the mainline EleutherAI `lm-evaluation-harness` under\n`lm_eval/tasks/swde`.\n\n## Lineage\n\nThe name traces back to the original SWDE (\"Structured Web Data Extraction\") corpus introduced by\nHao et al. at SIGIR 2011, 124,291 pages across 80 sites and eight verticals for wrapper-induction\nresearch, later extended with richer open-IE-style annotation by Lockard et al. (OpenCeres, NAACL\n2019). The lm-evaluation-harness task's own README instead credits Lockard et al. with originally\ncurating SWDE for open information extraction from the semi-structured web; that appears to\nconflate the original 2011 corpus with the later OpenCeres extension, since Hao et al.'s SIGIR 2011\npaper is the source cited elsewhere in the literature for the SWDE corpus itself. The\nlm-evaluation-harness `swde` task is a further, much smaller adaptation of that lineage \u2014\nrestricted to the movie vertical and reformatted as a zero-shot generation task for LLMs \u2014 via the\nHazyResearch `based-swde-v2` dataset built for the EVAPORATE paper and later reused in\nHazyResearch's \"recall-intensive\" evaluation suite for the BASED architecture line\n(arXiv:2402.18668). No successor or variant task of this specific harness adaptation exists in this\nrepository yet.\n\n## Saturation and contamination\n\nNo public leaderboard or standardized cross-model score table for this exact harness task was\nfound, so saturation status is unknown; scores reported in the EVAPORATE paper itself use a\ndifferent, non-harness evaluation pipeline and are not directly comparable to the harness's\n`contains` metric. Contamination risk is medium: the source pages are from a real public website and\nthe extraction dataset itself has been publicly available since 2023, so both the documents and\ntheir attribute values could plausibly appear in pretraining data independent of this benchmark.\n\n## How to run it\n\nRun via lm-evaluation-harness with `--tasks swde`; the task is implemented as a custom Python class\n(`task.py`'s `SWDE`, loaded through `swde.yaml`'s `class: !function task.SWDE`) rather than a plain\nYAML config, so behavior (prompt construction, target extraction, the `contains_score` metric) lives\nin that Python file. It is a zero-shot, generation-based task, not multiple choice. Because of the\n2026-06-22 whitespace-handling change noted in the README, confirm which harness version produced\nany number before comparing it to another report.\n\n## Reading the numbers\n\nA high `contains` score indicates a model can reliably locate and reproduce a short factual span\nfrom inside a long, noisy in-context document when told what attribute to look for \u2014 a narrow but\npractically relevant long-context retrieval skill. It says little about the model's ability to\ngeneralize extraction rules across previously unseen page layouts (the harder skill the original\nSWDE benchmark targeted), nor about structured-data extraction at the scale evaluated in the\nEVAPORATE paper itself. Because there is no established public leaderboard for this harness\nvariant, treat any single score as informative mainly in comparison to other models run under the\nidentical harness version.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swde",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Structured Web Data Extraction"
  ],
  "category": "long-context",
  "contamination": {
   "note": "The underlying page text and gold values are public (via the original SWDE corpus and the hazyresearch/based-swde-v2 Hugging Face dataset), and the movie-page content itself is drawn from a public website, so both the pages and plausible answer values could appear in pretraining data independent of this benchmark's own publication.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1111,
   "size_note": "The Hugging Face dataset card for hazyresearch/based-swde-v2 (the dataset this task loads) lists 1,111 rows drawn from 141 documents, in a single \"validation\" split, ~1.83 MB as parquet.\n",
   "splits": "validation only (1,111 rows / 141 documents); no separate train or test split in this harness dataset",
   "url": "https://huggingface.co/datasets/hazyresearch/based-swde-v2"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-007 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-007"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "swde",
   "opencompass": "",
   "other": ""
  },
  "id": "swde",
  "last_updated": "2026-06-22",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The model is given the raw text of a webpage (from the movie vertical of the SWDE web-extraction corpus) together with an attribute key such as \"release date,\" \"genre\" or \"director,\" and must produce the value for that attribute as it appears on the page. This exercises long-context associative recall and information extraction: the model must locate a specific fact inside a long, noisy, semi-structured document rather than answer from parametric knowledge. The lm-evaluation-harness README describes it as a version \"designed for the zero-shot evaluation of small language models,\" deliberately adapted to be somewhat easier than the original SWDE wrapper-induction formulation.\n",
  "metric": {
   "baseline_note": "lm-evaluation-harness scores each item with a \"contains\" metric (task.py's contains_score): a case-insensitive regex search for whether the gold value appears in the model's generated continuation, taking the max over any listed valid labels, then averaged across items with np.mean. No random or human baseline is published for this harness adaptation. The README notes that harness version 1 (2026-06-22) started stripping whitespace from prompts and targets, so scores from that version are not comparable to version 0.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "contains",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "SWDE (lm-evaluation-harness zero-shot extraction task)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2304.09433",
   "title": "Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes",
   "url": "https://arxiv.org/abs/2304.09433",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Simran Arora",
    "Brandon Yang",
    "Sabri Eyuboglu",
    "Avanika Narayan",
    "Andrew Hojel",
    "Immanuel Trummer",
    "Christopher R\u00e9"
   ],
   "org": "Stanford HazyResearch (dataset/task); EleutherAI (lm-evaluation-harness integration)",
   "url": "https://github.com/HazyResearch/based-evaluation-harness"
  },
  "released": "2023",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/swde",
  "saturation": {
   "as_of": "",
   "note": "No leaderboard or cross-model score table for this specific lm-evaluation-harness adaptation was located; the EVAPORATE paper reports its own extraction-quality numbers under a different (non-harness) evaluation protocol, so they are not directly comparable.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness: swde task directory",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/swde"
   },
   {
    "accessed": "2026-09-08",
    "title": "swde task README",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/swde/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "swde.yaml task config",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/swde/swde.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "swde task.py implementation",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/swde/task.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "hazyresearch/based-swde-v2 dataset card",
    "url": "https://huggingface.co/datasets/hazyresearch/based-swde-v2"
   },
   {
    "accessed": "2026-09-08",
    "title": "Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes (EVAPORATE)",
    "url": "https://arxiv.org/abs/2304.09433"
   },
   {
    "accessed": "2026-09-08",
    "title": "HazyResearch based-evaluation-harness",
    "url": "https://github.com/HazyResearch/based-evaluation-harness"
   }
  ],
  "status": "active",
  "subcategory": "in-context information extraction / associative recall from long HTML documents (movie vertical)",
  "summary": "Zero-shot task: given a full movie webpage's text in-context, extract the value for a named attribute such as release date or director.",
  "tags": [
   "information-extraction",
   "long-context",
   "recall",
   "html",
   "zero-shot"
  ],
  "task_format": "Zero-shot, free-form generation: prompt is page text + attribute key, target is the attribute's value as a short string."
 }
}