HELM wrap of CLEVA, a 31-task Chinese LLM platform with standardized prompts, 370K test instances, and contamination-aware sampling.
unassessed
| Category | composite |
|---|---|
| Subcategory | 31-task Chinese LLM suite (applications, abilities, and harms) with standardized prompts |
| Page status | active |
| Metric | task-dependent (exact match, Chinese ROUGE-2, Chinese BLEU, SacreBLEU, iBLEU, math exact match, and others) |
| Direction | higher_is_better |
| Dataset size | 370000 |
| Dataset licence | CC BY-NC-ND 4.0 |
| Publisher | CUHK LaVi Lab; Shanghai AI Laboratory (collaboration); HELM integration by Stanford CRFM |
cleva is HELM's scenario family for CLEVA (Chinese Language Models EVAluation Platform), not one quiz. The model is tested in Chinese across application tasks (for example translation, summarization, dialogue, code synthesis) and ability tasks (subject knowledge, reasoning primitives, calculation, cultural knowledge, harms). CLEVA's claim is that Chinese evals were incomparable because prompts were not shared. HELM therefore downloads CLEVA's per-task JSONL plus prompt templates and inference parameters, then applies the chosen template. Language is Chinese except where a task is explicitly bilingual (en2zh / zh2en).
HELM run-spec function cleva with args task, optional subtask, prompt_id (0-based template index), and version (v1 in the paper and README). Adaptation is generation, joint multiple choice, or separate multiple choice according to each prompt file's meta.mul_as_gen flag. Data are served from http://39.108.215.175/data as per-task zip files.
No model card in ModelSpec reports this benchmark yet.