HELM's Vietnamese synthetic-reasoning (natural) task: deduce attributes from generated Vietnamese rules and facts, scored by set-overlap F1.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | Vietnamese synthetic rule-fact deduction |
| Page status | unknown |
| Metric | f1_set_match |
| Direction | higher_is_better |
| Dataset size | 5000 |
| Publisher | Stanford CRFM (HELM MELT scenarios) |
MELT SRN generates Vietnamese rule-and-fact puzzles and asks the model to list what else must be true of a subject. Each item is a set of "if … then …" rules, one fact about a person, animal or plant, and a prompt asking what can be determined about that subject. The target is the set of attributes implied in one reasoning step. Easy mode always uses the specific subject (for example a name rather than "một người"). Hard mode substitutes more specific synonyms in the fact. The puzzles are synthetic Vietnamese, not mined text.
Open generation. HELM instruction "Hãy giải quyết vấn đề sau.", input noun "Quy luật", three in-context examples, max 20 tokens. Difficulty is a run-spec argument: easy or hard. Main metric is f1_set_match on the generated test split.
No model card in ModelSpec reports this benchmark yet.