HELM scenario that asks a model to name the one unit-test index a given C++ student is most likely to fail.
unassessed
| Category | coding |
|---|---|
| Subcategory | HELM prediction of which C++ unit test a CodeInsight student fails |
| Page status | unknown |
| Metric | unittest_alignment |
| Direction | higher_is_better |
| Dataset size | 23 |
| Publisher | Stanford CRFM (HELM scenario); CodeInsight dataset from VNU-HCM University of Technology CS |
codeinsights_edge_case is a Stanford HELM run spec on sampled CodeInsight logs. For each student it loads a topic pass-rate profile and one target C++ problem with parsed unit tests, then asks the model to output a single 0-based unit-test index the student would fail. Scoring extracts the first integer in the completion and checks it against the unique failing bit in that student's `pass` pattern. It is failure-index prediction, not code generation, and not the CodeInsight paper's next-attempt predictor.
English HELM instruction plus a Vietnamese/English problem and unit-test list; model should return an integer. Default temperature 0, max_tokens 4000, num_testcases=1 (only the first parsed tests are kept in extra_data). The generation adapter still uses output noun "Your code".
No model card in ModelSpec reports this benchmark yet.