HELM scenario that asks a model to fill a C++ course template so the generated body passes the problem's unit tests.
unassessed
| Category | coding |
|---|---|
| Subcategory | HELM C++ unit-test pass rate on a CodeInsight question sample |
| Page status | unknown |
| Metric | functional_correctness |
| Direction | higher_is_better |
| Dataset size | 84 |
| Publisher | Stanford CRFM (HELM scenario); CodeInsight dataset from VNU-HCM University of Technology CS |
codeinsights_correct_code is a Stanford HELM run spec on sampled CodeInsight course logs. It groups Kazchoko/my_dataset Scenario1_2_data.csv by question_unittest_id, takes the first row of each question, and asks the model to write only the C++ that replaces {{ STUDENT_ANSWER }} in the course template. Scoring compiles with g++ and measures the fraction of parsed unit tests that match expected stdout. It is functional correctness on undergraduate C++ assignments, not the CodeInsight paper's next-attempt predictor, not student-style imitation, and not [ci_mcqa](ci_mcqa.md).
English HELM instruction plus a Vietnamese/English problem prompt; model returns a fenced C++ fragment. Default temperature 0, max_tokens 4000, num_testcases=1 (only the first parsed unit test is kept).
No model card in ModelSpec reports this benchmark yet.