{
 "body": "\n## What it measures\n\n`medtext` is lm-eval's wrap of Hugging Face `BI55/MedText`. Each item is an English patient presentation and a gold diagnosis-and-treatment write-up. The model must produce that clinical plan from the presentation. The card covers common hospital diseases and injuries, plus noise cases where the model should refuse or ask for more information. The modality is text. The language is English.\n\nThis is not USMLE-style multiple choice and not MIMIC discharge summarization. The lm-eval README cites Melamud and Shivade 2019 on shareable synthetic clinical notes. That paper trains language models on de-identified notes. It is not this GPT-4 converted textbook-style set.\n\n## How it is scored\n\nThe generation task is `generate_until` with stop sequence `\\n\\n`. Scores are BLEU, ROUGE-1/2/L, BLEURT and BERTScore F1, each with nanmean. There is no YAML headline metric. Empty strings become NaN. `medtext_perplexity` reports word and byte perplexity and bits per byte on the completion. No random or official human score for this protocol was published. The card's three-doctor check of ten examples is a quality note, not a leaderboard baseline.\n\n## Dataset and licence\n\ndatasets-server lists 1,412 train rows. The csv `medtext_2.csv` matches that count. The card calls it the shuffled version of `medtext_1` and dates the dataset to 25 July 2023. lm-eval points train, validation and test at that same split, so a default run evaluates all 1,412 items with public golds. The licence on the card is CC BY 4.0. The card says the uniform datapoints were produced with GPT-4 from original text.\n\n## Who publishes it\n\nThe Hugging Face dataset is under user `BI55`, with profile fullname L. Heinrich. No separate academic paper for this csv was found. EleutherAI maintains the lm-eval task files. There is no official MedText leaderboard URL.\n\n## Lineage\n\nDo not treat this as Melamud and Shivade 2019, despite the lm-eval README citation. It is also not [medication_qa](medication_qa.md), [medi_qa](medi_qa.md), or the MIMIC discharge tasks [mimic_bhc](mimic_bhc.md) and [mimic_rrs](mimic_rrs.md). Those use real or gated clinical notes and different metrics. No successor replaced this wrap.\n\n## Saturation and contamination\n\nSaturation is unknown. Contamination risk is high: prompts and gold completions are public, small, and dated 2023. GPT-4 was used to format the pairs, so lexical overlap with other GPT-4 clinical text is possible even if this file was not copied.\n\n## How to run it\n\n```\nlm_eval --model hf --model_args pretrained=... --tasks medtext\n```\n\nAlso `medtext_perplexity`. Metric extras match the MEDIQA lm-eval helper (BLEU, ROUGE, BERTScore, BLEURT). Because every split is `train`, few-shot sampling can draw from the same 1,412 rows that are scored. Compare runs only when shot count and stop rules match.\n\n## Reading the numbers\n\nA high ROUGE or BERTScore means the model echoed this set's diagnosis-and-treatment style, including the humble \"cannot replace a doctor\" completions the card describes. It does not measure bedside accuracy, coding, or exam knowledge. Pair it with a held-out clinical QA or note task, and ignore the 2019 synthetic-note citation in the harness README.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "medtext",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "medtext_perplexity",
   "BI55/MedText"
  ],
  "category": "domain",
  "contamination": {
   "note": "The full 1,412 prompt-completion pairs have been public on Hugging Face since 25 July 2023, including the gold completions. The card says the text was converted into uniform datapoints with GPT-4, so models trained on similar GPT-4 clinical prose may overlap even without this file.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1412,
   "size_note": "Hugging Face datasets-server reports 1,412 train examples in BI55/MedText (csv medtext_2.csv). The downloaded csv has a header plus 1,412 rows. The card describes this file as the shuffled version of medtext_1. lm-eval uses that train split for training, validation and test, so a default run scores all 1,412 items. The card says \"over 1000\" presentations; the counted size is 1,412.\n",
   "splits": "single train split (1,412); lm-eval maps train/validation/test all to train",
   "url": "https://huggingface.co/datasets/BI55/MedText"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-058 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-058"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "medtext",
   "opencompass": "",
   "other": "Also medtext_perplexity (loglikelihood_rolling on the Completion field)."
  },
  "id": "medtext",
  "last_updated": "2023-07",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "This id is EleutherAI lm-evaluation-harness task medtext. The Hugging Face set BI55/MedText pairs an English patient presentation (Prompt) with a diagnosis and treatment completion written in a clinical voice. lm-eval asks the model to continue that presentation. It is text-only English generation. It is not a licensing exam, not MIMIC notes, and not the 2019 Melamud and Shivade synthetic clinical-note paper that the lm-eval README cites.\n",
  "metric": {
   "baseline_note": "lm-eval logs six overlap metrics with nanmean aggregation. BLEURT uses bleurt-base-512; BERTScore is English F1. The card says three doctors rated ten random examples as textbook quality; that is not a scored human baseline for this generation protocol. No random baseline is stated.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "bleu, rouge1, rouge2, rougeL, bleurt, bert_score (no designated headline)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "MedText (lm-eval)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [
    "L. Heinrich"
   ],
   "org": "Hugging Face user BI55 (card fullname L. Heinrich)",
   "url": "https://huggingface.co/datasets/BI55/MedText"
  },
  "released": "2023-07",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/medtext",
  "saturation": {
   "as_of": "",
   "note": "No public MedText leaderboard was opened. Saturation of the overlap metrics is not established.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness medtext task directory",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/medtext"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval medtext README (cites arXiv:1905.07002; does not match this dataset)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/medtext/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval medtext.yaml",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/medtext/medtext.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval medtext utils.py",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/medtext/utils.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "BI55/MedText dataset card (CC-BY-4.0, GPT-4 converted original data)",
    "url": "https://huggingface.co/datasets/BI55/MedText"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API dataset metadata (created 2023-07-25, 1412-row csv)",
    "url": "https://huggingface.co/api/datasets/BI55/MedText"
   },
   {
    "accessed": "2026-09-08",
    "title": "BI55/MedText datasets-server split counts (train 1412)",
    "url": "https://datasets-server.huggingface.co/info?dataset=BI55/MedText"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face user BI55 overview (fullname L. Heinrich)",
    "url": "https://huggingface.co/api/users/BI55/overview"
   },
   {
    "accessed": "2026-09-08",
    "title": "Melamud and Shivade 2019 (cited by lm-eval README; different work)",
    "url": "https://arxiv.org/abs/1905.07002"
   }
  ],
  "status": "unknown",
  "subcategory": "synthetic patient presentation to diagnosis and treatment",
  "summary": "lm-eval's generation wrap of BI55/MedText: write a diagnosis and treatment plan from an English patient presentation and score overlap metrics.",
  "tags": [
   "medical",
   "generation",
   "overlap-metrics",
   "lm-eval",
   "synthetic"
  ],
  "task_format": "generate_until generation, stopping at a blank line. doc_to_text is the Prompt field; gold is Completion. The YAML description instructs the model to answer as a doctor with a likely diagnosis and treatment. Train, validation and test all point at the single train split. Companion task medtext_perplexity scores rolling likelihood of the completion.\n"
 }
}