HELM runtime probe: generate from fixed public-domain prompts while varying prompt length, output length and tokenizer.
unassessed
| Category | generation |
|---|---|
| Subcategory | controlled inference-runtime probe over prompt length, output length and tokenizer |
| Page status | unknown |
| Metric | inference runtime (denoised and idealized); run spec also attaches exact_match |
| Direction | lower_is_better |
| Dataset size | 10 |
| Publisher | Stanford CRFM (HELM) |
synthetic_efficiency is not a quality ranking. HELM feeds short public-domain prompts of a chosen token length and asks the model to generate a chosen number of tokens so that runtime can be compared across providers. The scenario docstring lists four questions: how runtime scales with prompt and output length and with the number of completions; how much variance each query shows; how providers differ; and whether hardware can be reverse-engineered. Prompts are stored on a CodaLab bundle and sliced per tokenizer. English-script public-domain text used as filler, not as a comprehension test.
Completion generation. Run spec synthetic_efficiency with num_prompt_tokens, num_output_tokens, tokenizer and optional random. Each run loads 10 instances. References are empty strings tagged correct. HELM also attaches exact_match and generative-harms metrics even though the gold continuation is empty.
No model card in ModelSpec reports this benchmark yet.