A web-agent benchmark of 2,000-plus real-website tasks; the model picks the next HTML element and action (click, type, or select) on three held-out splits.
unassessed
| Category | agentic |
|---|---|
| Subcategory | real-website GUI / HTML action prediction |
| Page status | active |
| Metric | step success rate (also element accuracy, operation F1, task success rate) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2350 |
| Dataset licence | CC-BY-4.0 (osunlp/Mind2Web dump); Multimodal-Mind2Web is OpenRAIL; OSU-NLP-Group/Mind2Web code is MIT |
| Publisher | Ohio State University NLP Group (OSU-NLP-Group) |
Mind2Web gives a model a natural-language goal on a live-style webpage (for example, find a one-way flight) plus the HTML of the current page and the actions so far. It must choose the next element and operation: CLICK, TYPE, or SELECT, with a value when the operation needs one. The benchmark tests whether an agent can follow open-ended instructions on real websites rather than on simplified simulators. English text; the original dump also stores screenshots, but the inspect_evals port scores HTML multiple-choice action prediction, not pixels.
Two-stage MindAct setup from the paper: a DeBERTa ranker proposes candidate elements, then the model answers a multiple-choice question (option A is always "None of the above"; B–F are ranked elements) and names the operation. inspect_evals implements only the second stage, using cached ranker scores. Three test splits: Cross-Task, Cross-Website, Cross-Domain. Step-wise metrics are macro-averaged across tasks.
No model card in ModelSpec reports this benchmark yet.