HELM scenario that asks a model to introduce a given student's typical C++ mistakes on a new problem, using three prior buggy submissions.
unassessed
| Category | coding |
|---|---|
| Subcategory | HELM C++ student-mistake generation on a CodeInsight sample |
| Page status | unknown |
| Metric | composite: ast_distance, asm_distance, codebert_similarity, unittest_alignment_ratio |
| Direction | higher_is_better |
| Dataset size | 41 |
| Publisher | Stanford CRFM (HELM scenario); CodeInsight dataset from VNU-HCM University of Technology CS |
codeinsights_student_mistake is a Stanford HELM run spec on sampled CodeInsight course logs. For each retained student it shows a topic pass-rate profile and three earlier buggy C++ submissions, then asks the model to attempt a fourth problem while inserting mistakes that student would likely make. The instruction forbids a fully correct solution. HELM scores the generated fragment against the student's actual mistaken response and against that student's unit-test pass pattern. It is error-pattern imitation, not generic bug finding, not the CodeInsight paper's next-attempt predictor, and not [ci_mcqa](ci_mcqa.md).
English HELM instruction plus a Vietnamese/English problem prompt; model returns a fenced C++ fragment for {{ STUDENT_ANSWER }}. Default temperature 0, max_tokens 4000, num_testcases=1.
No model card in ModelSpec reports this benchmark yet.