HELM multiple-choice wrap that maps a short cyber threat intelligence sentence to a MITRE ATT&CK enterprise technique name.
unassessed
| Category | domain |
|---|---|
| Subcategory | multiple-choice mapping of CTI sentences to MITRE ATT&CK enterprise techniques |
| Page status | unknown |
| Metric | exact_match (HELM run spec; scenario metadata also names quasi_exact_match) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 12945 |
| Dataset licence | CC BY-SA 4.0 |
| Publisher | DIETI, Università degli Studi di Napoli Federico II, and University of Bern (HELM scenario by Stanford CRFM) |
HELM scenario cti_to_mitre turns Orbinato et al. 2022's CTI-to-technique classification into a generative multiple-choice task. The model sees one English sentence describing attacker behaviour and must pick the ATT&CK enterprise technique name from a lettered list. The source dataset labels sentences with technique IDs; HELM maps those IDs to names using MITRE ATT&CK v10.1 enterprise-attack.json. This is static text classification, not an agent writing detections. It is not [cti_realm](cti_realm.md).
Multiple-choice joint adaptation. HELM run spec default: num_options=10, seed=42, max_train_instances=10, instruction "Classify the following situation by the type of security attack. Answer with only a single letter." The last option is Others when the list is shorter than the full category set. Scenario code default num_options is 199 (all enterprise technique categories in the v10.1 zip), which is not the run-spec default.
No model card in ModelSpec reports this benchmark yet.