HELM scenario that asks a model to write C++ in one student's style given three of their submissions and a new problem.
unassessed
| Category | coding |
|---|---|
| Subcategory | HELM C++ student-style generation on a CodeInsight sample |
| Page status | unknown |
| Metric | composite: ast_distance, asm_distance, codebert_similarity, unittest_alignment_ratio |
| Direction | higher_is_better |
| Dataset size | 29 |
| Publisher | Stanford CRFM (HELM scenario); CodeInsight dataset from VNU-HCM University of Technology CS |
codeinsights_student_coding is a Stanford HELM run spec. It samples one undergraduate from a public CodeInsight CSV, shows three of that student's earlier C++ submissions, and asks the model to solve a fourth problem in the same personal style. Problem text is typically Vietnamese; the model must emit only a C++ body that drops into a {{ STUDENT_ANSWER }} template. HELM then compares that body to the student's own later submission and to unit-test outcomes. It is a style-and-behaviour imitation task, not the CodeInsight paper's next-attempt score prediction benchmark, and not [ci_mcqa](ci_mcqa.md).
English HELM instruction plus a Vietnamese/English problem prompt; model returns a fenced C++ fragment. Default run uses temperature 0, max_tokens 4000, and the first unit test only (num_testcases=1).
No model card in ModelSpec reports this benchmark yet.