{
 "body": "\n## What it measures\n\nPMC-Patients asks a retriever to help with a written case, not to diagnose it. The query is an English patient summary mined from a PubMed Central case report. Patient-to-article retrieval (PAR) must rank PubMed records that the citation graph treats as relevant. Patient-to-patient retrieval (PPR) must rank other summaries treated as similar patients.\n\nRelevance is 2 or 1 from citation distance, not a fresh clinician grade. PAR and PPR share the query patients and the train/dev/test split. The format matches [BEIR](beir.md) (queries, corpus, qrels) but this suite is not one of BEIR's 18 core datasets.\n\n## How it is scored\n\nThe paper and homepage use MRR, precision at 10, nDCG at 10, and recall at 1,000, reported as percentages. Official `evaluation.py` consumes a BEIR-style JSON of query-to-document scores. The live HTML table (retrieved 2026-09-08) still ranks DPR (SciMult-MHAExpert) first on PAR at 29.89 MRR / 9.35 P@10 / 13.79 nDCG@10 / 53.71 R@1k (25 June 2023). PPR is led by reciprocal rank fusion at 27.76 / 6.96 / 24.12 / 85.14 (5 April 2023). BM25 is sixth on PAR and third on PPR on that board.\n\n## Dataset and licence\n\nThe Nature descriptor (18 December 2023) counts 167,034 summaries that pass the filters (homepage and GitHub round to 167k), plus 3.1M article links and 293k patient links. Hugging Face `zhengyun21/PMC-Patients` and `zhengyun21/PMC-Patients-ReCDS` both tag CC-BY-NC-SA-4.0. The Nature article text is CC BY 4.0; the authors built the redistributable set only from PMC OA papers with at least CC BY-NC-SA. Test qrels are public. A 2024 V2 dump adds 250,294 patients and is not the ReCDS split.\n\nNature, the GitHub README, and the ReCDS card all put the PAR corpus at 11.7 million PubMed title/abstract records. arXiv 2202.13876 Table 1 listed 1.4 million candidate articles. Use the Nature/GitHub figure for the published benchmark.\n\n## Who publishes it\n\nZhengyun Zhao, Qiao Jin, Fangyuan Chen, Tuorui Peng, and Sheng Yu (Tsinghua). Scientific Data 10:909, received 3 July 2023, published 18 December 2023. Leaderboard submissions go by email to the contact on the GitHub README. Code and data: `github.com/pmc-patients/pmc-patients`. Collection code also sits at `zhao-zy15/PMC-Patients`.\n\n## Lineage\n\nThis is a clinical retrieval suite, not a QA set like [PubMedQA](pubmedqa.md) or [MedQA](medqa.md). It borrows BEIR packaging. It is unrelated to PMC-OA, which is a figure-caption pretraining dump. No successor page exists here. PMC-Patients-V2 enlarges the note collection; it does not replace the ReCDS leaderboard.\n\n## Saturation and contamination\n\nPAR MRR still sits near 30 on the public board, so the ceiling is open. The table looks stale: nothing newer than MedCPT-d (4 October 2023) appears in the HTML. Notes and labels have been public since 2023, so web-scale models may have seen them. Risk is medium.\n\n## How to run it\n\nDownload ReCDS from Hugging Face or Figshare, keep the `datasets` folder, dump ranked results as JSON, then `python evaluation.py --task PAR --split test --result_path ...` (or `PPR`). Submit to the homepage by email. No lm-eval, HELM, inspect_evals, or OpenCompass task name was found. Compare only runs that use the same PAR corpus size (11.7M vs the older 1.4M table).\n\n## Reading the numbers\n\nA 30 MRR on PAR means the first relevant article is still often far down a 11.7M-document list. That is citation-graph retrieval, not clinical correctness. P@10 under 10 on the leading PAR row shows that a short page of hits is usually mixed. Look at PPR separately: lexical fusion still wins that board. Do not treat V2's 250k notes as a new official test.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "pmc_patients",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "PMC-Patients ReCDS",
   "ReCDS-PAR",
   "ReCDS-PPR"
  ],
  "category": "embedding",
  "contamination": {
   "note": "Summaries and qrels are public (CC-BY-NC-SA-4.0). Source articles are PMC Open Access with at least CC BY-NC-SA. Labels come from the public citation graph. No publisher contamination study of LLM training data was found.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC-BY-NC-SA-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 167034,
   "size_note": "Nature text: 167,034 summaries pass the extraction filters; homepage and GitHub round this to 167k, from 141k PMC articles, with 3.1M patient-article links and 293k patient-patient links. ReCDS-PPR corpus is 155.2k reference patients. Nature, GitHub, and the ReCDS card give a PAR corpus of 11.7M PubMed articles. arXiv 2202.13876 Table 1 instead listed 1.4M candidate articles. HF also ships PMC-Patients-V2 with 250,294 patients (2024 PMC baseline); that dump is not the ReCDS split.\n",
   "splits": "train/dev/test queries shared by PAR and PPR; article-level split",
   "url": "https://huggingface.co/datasets/zhengyun21/PMC-Patients-ReCDS"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-079 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Official evaluation.py on github.com/pmc-patients/pmc-patients (BEIR-style JSON results). Not one of BEIR's 18 core datasets.\n"
  },
  "id": "pmc_patients",
  "last_updated": "2024-12",
  "leaderboard_url": "https://pmc-patients.github.io/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "PMC-Patients tests retrieval-based clinical decision support, not diagnosis from scratch. The query is an English patient summary taken from a PMC case report. PAR must rank PubMed title/abstract records that the citation graph marks relevant. PPR must rank other patient summaries marked similar. Labels are 2 or 1 from citation distance, not from a clinician re-reading the chart. Both tasks share the query split and use BEIR-style queries, corpus, and qrels.\n",
  "metric": {
   "baseline_note": "Official columns are MRR, P@10, nDCG@10, and R@1k (percent). The homepage PAR leader (25 Jun 2023) is DPR SciMult-MHAExpert at 29.89 / 9.35 / 13.79 / 53.71. The PPR leader (5 Apr 2023) is RRF at 27.76 / 6.96 / 24.12 / 85.14. BM25 remains a strong PPR baseline on that board.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "MRR",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "PMC-Patients",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2202.13876",
   "title": "A large-scale dataset of patient summaries for retrieval-based clinical decision support systems",
   "url": "https://www.nature.com/articles/s41597-023-02814-8",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Zhengyun Zhao",
    "Qiao Jin",
    "Fangyuan Chen",
    "Tuorui Peng",
    "Sheng Yu"
   ],
   "org": "Tsinghua University",
   "url": "https://pmc-patients.github.io/"
  },
  "released": "2023-12",
  "repo_url": "https://github.com/pmc-patients/pmc-patients",
  "saturation": {
   "as_of": "2023-06",
   "note": "Homepage PAR MRR 29.89 (DPR SciMult-MHAExpert, 25 Jun 2023). PPR MRR 27.76 (RRF, 5 Apr 2023). Last listed run is MedCPT-d on 4 Oct 2023. Ceiling is not in reach. The board has not added rows in the HTML retrieved 2026-09-08.\n",
   "status": "open",
   "top_score": 29.89
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "PMC-Patients homepage and PAR/PPR leaderboards",
    "url": "https://pmc-patients.github.io/"
   },
   {
    "accessed": "2026-09-08",
    "title": "Scientific Data article (published 18 Dec 2023, CC BY 4.0 article)",
    "url": "https://www.nature.com/articles/s41597-023-02814-8"
   },
   {
    "accessed": "2026-09-08",
    "title": "arXiv 2202.13876 full text (metrics and Table 1)",
    "url": "https://ar5iv.labs.arxiv.org/html/2202.13876"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official PMC-Patients GitHub README",
    "url": "https://raw.githubusercontent.com/pmc-patients/pmc-patients/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "zhengyun21/PMC-Patients card (CC-BY-NC-SA-4.0, V2 note)",
    "url": "https://huggingface.co/datasets/zhengyun21/PMC-Patients/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "PMC-Patients Hub API (license tag cc-by-nc-sa-4.0)",
    "url": "https://huggingface.co/api/datasets/zhengyun21/PMC-Patients"
   },
   {
    "accessed": "2026-09-08",
    "title": "ReCDS benchmark card (11.7M PAR corpus; 155.2k PPR)",
    "url": "https://huggingface.co/datasets/zhengyun21/PMC-Patients-ReCDS/raw/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "ReCDS Hub API",
    "url": "https://huggingface.co/api/datasets/zhengyun21/PMC-Patients-ReCDS"
   }
  ],
  "status": "active",
  "subcategory": "patient-to-article and patient-to-patient clinical retrieval",
  "summary": "167k PMC case-report summaries with citation-graph labels for retrieving relevant PubMed articles (PAR) and similar patients (PPR).\n",
  "tags": [
   "biomedical",
   "retrieval",
   "clinical",
   "english",
   "beir-format"
  ],
  "task_format": "English text retrieval. Query: patient summary. PAR corpus: PubMed title+abstract. PPR corpus: other patient notes. Train/dev/test splits.\n"
 }
}