Zero-shot extraction of labelled values from chunked FDA 510(k) PDFs; lm-eval scores whether the gold value appears in a short continuation.
unassessed
| Category | domain |
|---|---|
| Subcategory | zero-shot key-value extraction from FDA 510(k) PDF chunks |
| Page status | unknown |
| Metric | contains |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1102 |
| Publisher | Hazy Research (Stanford University and collaborators) |
fda is not a drug-approval exam and not an FDA-wide leaderboard. It is the information-extraction slice that Arora et al. 2024 (BASED) built on the FDA 510(k) PDFs labelled in Arora et al. 2023 (EVAPORATE). The model sees a text chunk from a premarket-notification review plus a field name, and must continue with the value of that field. English text only. The original EVAPORATE setting used 100 PDFs (up to about 20 pages) and 16 gold attributes sampled from FDA 510(k) reviews since 1996. BASED and lm-eval score chunked prompts, not whole-document OpenIE.
Zero-shot generate-until. Prompt template is "{chunk}\\n{key}:" after stripping surrounding whitespace from the Hugging Face `text` field. The model may emit at most 48 tokens, stopping at a newline. Scoring is case-insensitive substring match of the stripped gold `value`.
No model card in ModelSpec reports this benchmark yet.