242 GPT-4-generated multiple-choice questions on computer security, in two difficulty tiers (v1: 127, v2: 115), scored on accuracy.
unassessed
| Category | domain |
|---|---|
| Subcategory | computer security knowledge, multiple choice |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 242 |
| Dataset licence | CC BY-NC-SA 4.0 |
| Publisher | Georgia Institute of Technology |
SecQA tests understanding of computer security principles: access control, cryptography basics, network security, and related topics drawn from a security-planning textbook. Questions are four-option multiple choice. v1 covers foundational, introductory-level concepts; v2 raises the difficulty with more nuanced and applied scenarios. Each item has a short rationale attached in the source dataset, though the evaluation itself only grades the selected option.
Four-option multiple choice, graded on the single correct letter. inspect_evals implements four task variants: sec_qa_v1, sec_qa_v1_5_shot, sec_qa_v2 and sec_qa_v2_5_shot, covering zero-shot and five-shot prompting for each difficulty tier.
No model card in ModelSpec reports this benchmark yet.