A HELM scenario for CS-education concept-inventory multiple-choice questions that cannot be run outside its authors because the item data is private.
unassessed
| Category | domain |
|---|---|
| Subcategory | CS education concept-inventory multiple-choice questions |
| Page status | unknown |
| Metric | exact match (single-letter multiple-choice accuracy) |
| Direction | higher_is_better |
| Unit | % |
ci_mcqa is the internal name HELM (Stanford's Holistic Evaluation of Language Models) uses for a scenario class called CIMCQAScenario, registered as the run spec "ci_mcqa". The scenario's own docstring describes it as "a multiple-choice question answering (MCQA) dataset designed to study concept inventories in CS Education." A concept inventory is a standardized instrument used in education research to detect specific, well-documented misconceptions a learner holds about a subject; the format originated in physics education and has since been adapted to other fields, including, here, computer science. Each item is a short question with several lettered answer options and exactly one option marked correct in the source data.
Multiple-choice question answering; HELM's adapter instructs the model to answer with a single letter chosen from the options given. The scenario code reads a separate test split and a (normally disabled) few-shot training split from local files.
No model card in ModelSpec reports this benchmark yet.