MedHELM's private counseling task: generate the next English counselor turn from dialogue history and score it with an LLM jury.
unassessed
| Category | domain |
|---|---|
| Subcategory | counseling-response generation (LLM jury) |
| Page status | active |
| Metric | mental_health_accuracy (HELM LLM-jury average of accuracy, completeness, clarity, each 1-5) |
| Direction | higher_is_better |
| Unit | points |
| Publisher | Stanford CRFM and collaborators (MedHELM) |
This id is HELM's mental_health scenario in MedHELM, not a public therapy dataset and not a psychiatric diagnostic exam. The model reads an English conversation history labelled with counselor and client turns, plus topic and dialogue-type fields, and must write the next counselor response. HELM scores that text with an LLM jury on accuracy, completeness and clarity. Inputs and outputs are English text.
Generation. Instruction: "Given a mental health conversation history, generate an empathetic and appropriate counselor response." Output noun Counselor response. max_tokens 512 in the run spec; some MedHELM private entries raise num_output_tokens to 4000. The scenario emits only TEST_SPLIT. get_generation_adapter_spec defaults to five train instances, but no train split exists, so the protocol is effectively zero-shot.
No model card in ModelSpec reports this benchmark yet.