{
 "body": "\n## What it measures\n\nSecQA checks whether a model can answer multiple-choice questions about computer security: concepts like access control models, cryptographic primitives, network defence and security planning. The items were generated by GPT-4 from the textbook \"Computer Systems Security: Planning for Success,\" then organised into two tiers. v1 targets foundational understanding; v2 raises the bar with more complex, applied scenarios. Every question carries a short rationale in the source data, though standard scoring only checks the selected letter, not the reasoning.\n\nThe benchmark is narrow by design (\"concise\" is in the paper's own title): 242 items total, not a broad security curriculum. It is meant as a quick signal of security-domain competence, not a substitute for a red-team or CTF-style practical evaluation.\n\n## How it is scored\n\nEach question has four options and one correct answer; the metric is plain accuracy over the datapoints attempted. inspect_evals exposes four separate tasks so that difficulty (v1 vs v2) and shot count (zero-shot vs the \"5_shot\" variants) can be reported independently rather than folded into one number. Because there are four options, random guessing scores 25% in expectation; no human-performance baseline was located in the paper or dataset card.\n\n## Dataset and licence\n\n127 questions make up v1 and 115 make up v2, for 242 total, per the paper and corroborating summaries; the Hugging Face dataset card and its datasets-server splits endpoint confirm two configs (secqa_v1, secqa_v2) each broken into dev, val and test, but did not return exact per-split row counts when checked for this page. The dataset card states a CC BY-NC-SA 4.0 licence. Test-split answers are shipped publicly, so this is not a held-out set.\n\n## Who publishes it\n\nThe paper lists a single author, Zefang Liu, submitted to arXiv in December 2023 and affiliated with the Georgia Institute of Technology per the paper's listed affiliation. The dataset is hosted on Hugging Face under `zefang-liu/secqa`, and the evaluation implementation used by this page is the UK AI Security Institute's inspect_evals repository (`UKGovernmentBEIS/inspect_evals`), which is the harness maintainer rather than the original publisher. No dedicated project leaderboard was found.\n\n## Lineage\n\nSecQA is a standalone 2023 security-knowledge multiple-choice set; no predecessor, successor or variant benchmark inside this repository was identified as being derived from it. It sits alongside other narrow domain-MCQ benchmarks (e.g. medical or legal MCQ sets) rather than a symbolic-reasoning or coding-security family; it should not be confused with broader cybersecurity capability benchmarks that test exploit writing or CTF solving, which SecQA does not attempt.\n\n## Saturation and contamination\n\nThe original December 2023 evaluation covered GPT-3.5-Turbo, GPT-4, Llama-2, Vicuna, Mistral and Zephyr in zero-shot and five-shot settings; specific scores were not extracted from a source opened for this page, and no current public leaderboard tracking frontier models on SecQA was found, so saturation status is unknown rather than assumed. Contamination risk is medium: the questions are public with answers, the source textbook is a named commercial publication rather than a private corpus, and the dataset predates the training cutoffs of most models evaluated on it today.\n\n## How to run it\n\ninspect_evals ships the benchmark as four ready-to-run tasks (`sec_qa_v1`, `sec_qa_v1_5_shot`, `sec_qa_v2`, `sec_qa_v2_5_shot`); each loads directly from `zefang-liu/secqa` on Hugging Face and grades with inspect's built-in multiple-choice scorer. No lm-evaluation-harness, HELM, OpenCompass or BIG-bench implementation was found. Because zero-shot and five-shot variants are separate tasks, a reported \"SecQA\" number should specify shot count and version (v1 vs v2) to be comparable across reports.\n\n## Reading the numbers\n\nA high SecQA accuracy shows a model can pick the textbook-correct answer to a security multiple-choice question, evaluated zero- or few-shot. Given only 242 questions split across two tiers, a handful of items can move the score by a percentage point or more, so treat close scores between models as noise rather than a real ranking. The benchmark does not test applied skills like writing an exploit, reading a log, or reasoning about a live system; pair it with a hands-on security evaluation before drawing conclusions about real-world security capability.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "sec_qa",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "sec_qa",
   "SecQA v1",
   "SecQA v2"
  ],
  "category": "domain",
  "contamination": {
   "note": "Test-split answers are public on Hugging Face and the questions were generated by GPT-4 from a named, purchasable textbook rather than held out by the publisher, so both the textbook source and the dataset itself are plausible pretraining-corpus contents for models trained after December 2023.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "CC BY-NC-SA 4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 242,
   "size_note": "Paper (arXiv:2312.15838) and secondary sources describe SecQA v1 as 127 questions and v2 as 115 questions, summing to 242. The Hugging Face dataset zefang-liu/secqa exposes two configs, secqa_v1 and secqa_v2, each split into dev, val and test; per-split row counts were not resolved from the datasets-server splits endpoint at the time this page was written.\n",
   "splits": "dev / val / test, separately for secqa_v1 and secqa_v2",
   "url": "https://huggingface.co/datasets/zefang-liu/secqa"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "sec_qa",
   "lm_eval": "",
   "opencompass": "",
   "other": "Task names in inspect_evals: sec_qa_v1, sec_qa_v1_5_shot, sec_qa_v2, sec_qa_v2_5_shot."
  },
  "id": "sec_qa",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "SecQA tests understanding of computer security principles: access control, cryptography basics, network security, and related topics drawn from a security-planning textbook. Questions are four-option multiple choice. v1 covers foundational, introductory-level concepts; v2 raises the difficulty with more nuanced and applied scenarios. Each item has a short rationale attached in the source dataset, though the evaluation itself only grades the selected option.\n",
  "metric": {
   "baseline_note": "inspect_evals' README states \"a simple accuracy is calculated over the datapoints.\" Random baseline of 25% follows from four options per question; no human baseline was read from the paper or repository.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": 25.0,
   "unit": "%"
  },
  "name": "SecQA",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2312.15838",
   "title": "SecQA: A Concise Question-Answering Dataset for Evaluating Large Language Models in Computer Security",
   "url": "https://arxiv.org/abs/2312.15838",
   "year": 2023
  },
  "publisher": {
   "authors": [
    "Zefang Liu"
   ],
   "org": "Georgia Institute of Technology",
   "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/sec_qa"
  },
  "released": "2023-12",
  "repo_url": "https://huggingface.co/datasets/zefang-liu/secqa",
  "saturation": {
   "as_of": "",
   "note": "The original paper evaluated GPT-3.5-Turbo, GPT-4, Llama-2, Vicuna, Mistral and Zephyr in 0-shot and 5-shot settings in December 2023, but a specific top score was not extracted from a source opened for this page. No current public leaderboard was found, so status is left unknown rather than assumed saturated on a 242-item multiple-choice set that is more than two years old relative to frontier models.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SecQA paper abstract (arXiv:2312.15838), single-author, GPT-4-generated MCQs from a security textbook",
    "url": "https://arxiv.org/abs/2312.15838"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals sec_qa README (task names, v1/v2 description, accuracy metric)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/sec_qa/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "zefang-liu/secqa dataset card (CC-BY-NC-SA-4.0 licence, secqa_v1/v2 configs)",
    "url": "https://huggingface.co/datasets/zefang-liu/secqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset API for zefang-liu/secqa (license tag, task_categories)",
    "url": "https://huggingface.co/api/datasets/zefang-liu/secqa"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face datasets-server splits endpoint (secqa_v1/secqa_v2, dev/val/test)",
    "url": "https://datasets-server.huggingface.co/splits?dataset=zefang-liu/secqa"
   }
  ],
  "status": "active",
  "subcategory": "computer security knowledge, multiple choice",
  "summary": "242 GPT-4-generated multiple-choice questions on computer security, in two difficulty tiers (v1: 127, v2: 115), scored on accuracy.",
  "tags": [
   "security",
   "cybersecurity",
   "multiple-choice",
   "domain-knowledge"
  ],
  "task_format": "Four-option multiple choice, graded on the single correct letter. inspect_evals implements four task variants: sec_qa_v1, sec_qa_v1_5_shot, sec_qa_v2 and sec_qa_v2_5_shot, covering zero-shot and five-shot prompting for each difficulty tier.\n"
 }
}