HELM Instruct scenario that scores 252 expert-written Self-Instruct tasks with a 1-5 Helpfulness critique, not the 52k generated training set.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | HELM Instruct critique of 252 user-oriented Self-Instruct instructions |
| Page status | unknown |
| Metric | Helpfulness |
| Direction | higher_is_better |
| Unit | 1-5 |
| Dataset size | 252 |
| Dataset licence | Apache-2.0 |
| Publisher | University of Washington and collaborators (dataset); Stanford CRFM (HELM scenario) |
HELM's self_instruct scenario loads the 252 expert-written, user-oriented instructions released for human evaluation in Wang et al.'s Self-Instruct paper. The model sees the instruction plus any instance input and must write a free-form English response. The scenario docstring states HELM is not running the Self-Instruct bootstrapping method and is not using the 52k model-written training instructions. The skill is following a novel user-style request (rewrite, tone, planning, and similar), not a closed NLP label set.
Zero-shot generation. HELM concatenates `instruction`, a newline, and `instances[].input`. Adapter defaults from `get_instruct_adapter_spec`: max_tokens 512, temperature 0.7, empty instruction prefix, `max_train_instances` 0. Scenario class `.name` is `self-instruct`; run spec and metadata name are `self_instruct`. All instances are tagged TEST_SPLIT.
No model card in ModelSpec reports this benchmark yet.