HELM Instruct scenario over OASST1 initial prompts, scored with a 1-5 Helpfulness critique rather than gold replies.
unassessed
| Category | instruction-following |
|---|---|
| Subcategory | HELM Instruct critique of OASST1 first-turn prompts |
| Page status | unknown |
| Metric | Helpfulness |
| Direction | higher_is_better |
| Unit | 1-5 |
| Dataset size | 88838 |
| Dataset licence | Apache-2.0 |
| Publisher | LAION / Open Assistant (dataset); Stanford CRFM (HELM scenario) |
open_assistant, as this id, is Stanford CRFM's OpenAssistantScenario. The model sees the root prompter message from an OASST1 conversation tree and writes a free-form reply. HELM keeps first-turn assistant messages as references but scores with InstructionFollowingCritiqueMetric, not overlap against those replies. OASST1 is LAION's crowd-sourced assistant corpus in 35 languages. Later turns in the tree are unused. This is not [koala](koala.md) or [self_instruct](self_instruct.md).
Zero-shot generation. Run spec open_assistant:language={lang} with lang a Hub language code or all. Adapter from get_instruct_adapter_spec: max_tokens 512, temperature 0.7, max_train_instances 0. Loads OpenAssistant/oasst1 at revision fdf72ae0827c1cda404aff25b6603abec9e3399b. Roots (parent_id is None) become inputs; child assistant texts become CORRECT_TAG references. Train and validation Hub splits are both loaded; metadata main_split is valid.
No model card in ModelSpec reports this benchmark yet.