Bilingual English/Chinese cyber-threat-intelligence bench of 1,300 test items: 100 four-way MCQs and 1,200 free-form QA items on incident analysis.
unassessed
| Category | domain |
|---|---|
| Subcategory | bilingual English/Chinese cyber threat intelligence: incident understanding MCQ and generation QA |
| Page status | active |
| Metric | MCQ accuracy (choice); QA ROUGE-L (0–100) and semantic cosine similarity |
| Direction | higher_is_better |
| Dataset size | 1300 |
| Dataset licence | Apache-2.0 |
| Publisher | Beihang University and collaborators (Multilingual-Multimodal-NLP) |
SEvenLLM-Bench tests whether a model can analyse cybersecurity incidents in English and Simplified Chinese. Understanding items are four-way multiple choice. Generation items ask for free-form answers such as key-entity lists, malware features, or attack-strategy writeups. The paper’s instruction corpus covers 28 expert-vetted CTI tasks over crawled incident text. Inspect Evals splits the public test dump into four runs: MCQ and QA, each in zh and en.
Inspect tasks wrap Hugging Face test.jsonl at revision 1de23ce. MCQ uses a fixed instruction template and Inspect multiple_choice with choice() scoring. QA uses generate() plus ROUGE-L (0.2 sentence threshold, jieba for Chinese) and cosine similarity from paraphrase-multilingual-MiniLM-L12-v2. The authors’ own code also reports GPT-4 scores and human ratings, which Inspect does not implement.
No model card in ModelSpec reports this benchmark yet.