Zero-shot task: given a full movie webpage's text in-context, extract the value for a named attribute such as release date or director.
unassessed
| Category | long-context |
|---|---|
| Subcategory | in-context information extraction / associative recall from long HTML documents (movie vertical) |
| Page status | active |
| Metric | contains |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1111 |
| Publisher | Stanford HazyResearch (dataset/task); EleutherAI (lm-evaluation-harness integration) |
The model is given the raw text of a webpage (from the movie vertical of the SWDE web-extraction corpus) together with an attribute key such as "release date," "genre" or "director," and must produce the value for that attribute as it appears on the page. This exercises long-context associative recall and information extraction: the model must locate a specific fact inside a long, noisy, semi-structured document rather than answer from parametric knowledge. The lm-evaluation-harness README describes it as a version "designed for the zero-shot evaluation of small language models," deliberately adapted to be somewhat easier than the original SWDE wrapper-induction formulation.
Zero-shot, free-form generation: prompt is page text + attribute key, target is the attribute's value as a short string.
No model card in ModelSpec reports this benchmark yet.