HELM Instruct scenario over 180 Koala user prompts, scored with a 1-5 Helpfulness critique rather than gold answers.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | HELM Instruct critique of 180 Koala user prompts |
| Page status | unknown |
| Metric | Helpfulness |
| Direction | higher_is_better |
| Unit | 1-5 |
| Dataset size | 180 |
| Dataset licence | Apache-2.0 (test-set LICENSE file); README additionally asks that the set not be used for training |
| Publisher | Berkeley Artificial Intelligence Research Lab (BAIR), UC Berkeley (test set); Stanford CRFM (HELM scenario) |
koala, as this id, is HELM's KoalaScenario: the 180 English user prompts the BAIR Koala team used to human-evaluate their dialogue model. The model sees only the prompt and writes a free-form reply. There is no gold reference in the HELM instances. The skill is following a real web user's request (the test-set README says non-English and coding prompts were removed so raters could judge them). This is not the Koala-13B training mix, not Alpaca's 180 self-instruct eval queries, and not [self_instruct](self_instruct.md)'s 252 tasks.
Zero-shot generation. HELM downloads koala_test_set.jsonl and emits one TEST_SPLIT Instance per line with empty references. Adapter defaults from get_instruct_adapter_spec: max_tokens 512, temperature 0.7, max_train_instances 0. Run spec name is koala. HELM Instruct's blog states that scenarios with more than 100 instances are randomly sampled to 100 in that study.
No model card in ModelSpec reports this benchmark yet.