HELM scenario that asks a model to write C++ matching one student's runtime style, then compares wall-clock times on unit tests.
unassessed
| Category | coding |
|---|---|
| Subcategory | HELM C++ runtime alignment against a CodeInsight student sample |
| Page status | unknown |
| Metric | composite: functional_correctness, runtime_efficiency_ratio, efficiency_alignment_score |
| Direction | higher_is_better |
| Dataset size | 19 |
| Publisher | Stanford CRFM (HELM scenario); CodeInsight dataset from VNU-HCM University of Technology CS |
codeinsights_code_efficiency is a Stanford HELM run spec on sampled CodeInsight logs. For each retained student it shows three earlier C++ submissions and a fourth Vietnamese problem, and asks the model to solve that problem in the same personal style, including the student's inefficiency when the examples are slow. Scoring compiles both the model body and the student's target response, then compares mean runtime on tests that pass. It is runtime imitation, not generic Big-O ranking, and not the CodeInsight paper's next-attempt predictor.
English HELM instruction plus a Vietnamese/English problem prompt; model returns a fenced C++ fragment for {{ STUDENT_ANSWER }}. Default temperature 0, max_tokens 4000, num_testcases=1. Efficiency metric uses 5 timed runs and a 10 s timeout.
No model card in ModelSpec reports this benchmark yet.