{
 "body": "\n## What it measures\n\nThe OpenCompass `nejmaibench` configuration is a medical multiple-choice evaluation tied to the nephSAP nephrology study. The configuration exposes a question, answer options, subject, and prompt mode, and uses a medical-assistant system prompt.\n\nThe configured zero-shot prompt asks the model to output only the letter corresponding to the correct answer. The linked paper identifies nephrology as the clinical domain and reports 858 Nephrology Self-Assessment Program questions.\n\n## How it is scored\n\nOpenCompass uses `NejmaibenchEvaluator` with generated answers and a zero-shot retriever. The evaluator extracts option letters and reports accuracy. The paper reports historical zero-shot scores, including GPT-4 at 73.3% and Claude 2 at 54.4%; these are paper results rather than a current leaderboard.\n\n## Dataset and licence\n\nThe OpenCompass data path is `opencompass/nejmaibench`, backed by a local `NEJM_All_Questions_And_Answers.csv` path in the project data registry. The linked paper reports 858 nephSAP questions. A public dataset licence and exact CSV revision were not established, so answer exposure remains unknown.\n\n## Who publishes it\n\nOpenCompass publishes the runnable configuration. Wu and co-authors published the linked nephSAP study in 2023. No current standalone leaderboard was established.\n\n## Lineage\n\nThis page documents the OpenCompass `nejmaibench` wrapper for the nephSAP nephrology question set. It should not be merged with broad medical exams or image-based NEJM benchmarks. No predecessor or successor was established.\n\n## Saturation and contamination\n\nSaturation and contamination risk are unknown. Medical questions may overlap with training data, but the inspected sources provide no exposure analysis. Scores should be interpreted with the prompt and dataset revision recorded.\n\n## How to run it\n\nRun the OpenCompass `nejmaibench` dataset configuration. It uses `NejmaibenchDataset`, `NejmaibenchEvaluator`, `ZeroRetriever`, and `GenInferencer`; it supplies the medical-assistant system prompt and asks for one option letter. Other prompt modes in the repository may produce different results.\n\n## Reading the numbers\n\nA strong score indicates success on these nephrology questions under the exact option format. It does not establish safe clinical decision-making, current medical knowledge, or performance with patient context. Check evaluator normalization, dataset revision, and whether answers were exposed before comparing reports.\n\nMedical multiple-choice performance can also depend on specialty mix and the distinction between recall and clinical reasoning. Those dimensions are not recoverable from the OpenCompass configuration alone, so they should be reported from the dataset release when available.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "nejm_ai_benchmark",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Nejmaibench",
   "nephSAP"
  ],
  "category": "domain",
  "contamination": {
   "note": "Exposure and contamination were not established.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": 858,
   "size_note": "The linked nephSAP paper reports 858 multiple-choice questions; the OpenCompass local CSV revision is not independently counted here.",
   "splits": "",
   "url": "https://huggingface.co/datasets/opencompass/nejmaibench"
  },
  "freshness": {
   "luna-batch-017": null,
   "luna-batch-017 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "nejmaibench",
   "other": ""
  },
  "id": "nejm_ai_benchmark",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The OpenCompass configuration identifies a medical question, answer options, subject, and prompt mode. The linked benchmark paper evaluates 858 Nephrology Self-Assessment Program questions, not a general NEJM exam.",
  "metric": {
   "direction": "higher_is_better",
   "max_score": 100,
   "name": "accuracy",
   "unit": "percent"
  },
  "name": "NEJMAI / nephSAP nephrology benchmark",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2308.04709",
   "title": "A Comparative Study of Open-Source Large Language Models, GPT-4 and Claude 2: Multiple-Choice Test Taking in Nephrology",
   "url": "https://arxiv.org/abs/2308.04709",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Sean Wu",
    "Michael Koo",
    "Lesley Blum",
    "Andy Black",
    "Liyo Kao",
    "Fabien Scalzo",
    "Ira Kurtz"
   ],
   "org": "Nephrology Self-Assessment Program / OpenCompass",
   "url": "https://arxiv.org/abs/2308.04709"
  },
  "released": "2023-08",
  "repo_url": "https://github.com/open-compass/opencompass",
  "saturation": {
   "as_of": "",
   "note": "No authoritative leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass NEJMAI benchmark configuration",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/nejm_ai_benchmark/nejmaibench_gen_60c8f5.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass NEJMAI dataset card",
    "url": "https://huggingface.co/datasets/opencompass/nejmaibench"
   }
  ],
  "status": "active",
  "subcategory": "medical knowledge",
  "summary": "OpenCompass's NEJMAI wrapper evaluates zero-shot multiple-choice answering on 858 nephSAP nephrology questions.",
  "tags": [
   "medicine",
   "multiple-choice",
   "zero-shot"
  ],
  "task_format": "Medical multiple-choice question with options A through E; output one option letter."
 }
}