Four public cybersecurity multiple-choice sets (80, 500, 2,000, 10,000 items) built with RAG and expert review to test LLM security knowledge.
unassessed
| Category | domain |
|---|---|
| Subcategory | cybersecurity four-option knowledge QA (80 / 500 / 2,000 / 10,000) |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 10000 |
| Publisher | Technology Innovation Institute (TII), University of Oslo, and Khalifa University |
CyberMetric asks a model to pick A, B, C, or D on English questions about cybersecurity practice. inspect_evals lists nine topic buckets: disaster recovery and BCP, IAM, IoT security, cryptography, wireless, network, cloud, penetration testing, and compliance/audit. Items were drafted with RAG over standards, certifications, papers, and books, then checked by people. The 80-item file is the human-comparison set; the three larger files are separate published sizes, not proven nested subsets.
Four-option multiple choice. inspect_evals uses system message "You are a security expert who answers questions." plus Inspect's multiple_choice solver and choice() scorer, expecting ANSWER: LETTER. The authors' CyberMetric_evaluator.py uses the same system line and an ANSWER: X regex. Their README also shows an XML <xml>D</xml> prompt used with some instruction-tuned models.
No model card in ModelSpec reports this benchmark yet.