lm-eval wrap of MedLFQA: English biomedical long answers scored with BLEU, ROUGE, BERTScore and BLEURT on a 10% slice.
unassessed
| Category | domain |
|---|---|
| Subcategory | biomedical long-form question answering |
| Page status | active |
| Metric | BLEU, ROUGE-1/2/L, BERTScore, BLEURT (nanmean) |
| Direction | higher_is_better |
| Dataset size | 4948 |
| Dataset licence | CC-BY-4.0 |
| Publisher | Korea University DMIS Lab (with Upstage AI and AIGEN Sciences) |
olaph, as this id, is EleutherAI's generate-until task on dmis-lab/MedLFQA. The model reads an English patient-style biomedical question and must write a long answer. Jeong et al. built MedLFQA so they could score factual claims (Must Have / Nice to Have statements) while training the OLAPH preference loop. The harness task does not score those claims. It compares the free-form answer to Free_form_answer with n-gram and embedding overlap. English text only. This is not [HealthBench](healthbench.md).
Open-ended generation. YAML description tells the model it is a healthcare assistant and to answer concisely without omitting relevant information. Stops at a blank line. training/validation/test splits all point at the Hub test split. process_docs keeps the first 10% of rows (code uses 0.1; the comment says 1%). Sister task olaph_perplexity uses loglikelihood_rolling on the same slice.
No model card in ModelSpec reports this benchmark yet.