Inspect AI agent benchmark where a model writes Sigma and KQL detection rules from cyber threat intelligence reports against live Kusto telemetry.
unassessed
| Category | agentic |
|---|---|
| Subcategory | agentic detection-rule writing from CTI reports against Kusto telemetry |
| Page status | active |
| Metric | weighted trajectory reward R_total in [0, 1] |
| Direction | higher_is_better |
| Dataset size | 50 |
| Dataset licence | MIT (inspect_evals package and CTI_REALM_TRANSPARENCY.md); Hugging Face dataset licence not stated |
| Publisher | Microsoft Security AI (inspect_evals packaging by UK AI Security Institute) |
CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) tests whether an AI agent can turn a detection objective and public CTI reports into working detections. The agent must find a relevant report, map MITRE ATT&CK techniques, explore Kusto tables, write and run KQL, and emit a Sigma rule plus query results as JSON. The environment is a Docker stack with a shared Kusto emulator, 12 log sources, 37 CTI reports, and cached MITRE and Sigma stores. Attacks are emulated on Linux endpoints, Azure Kubernetes Service (AKS), and Azure cloud. Scoring is a five-checkpoint trajectory reward in [0, 1], not a static multiple-choice grade. This page treats it as agentic because success depends on tool use against live telemetry.
ReAct agent, English. inspect_evals defaults: message_limit 70, one epoch, hard difficulty (minimal workflow hints). Output must be JSON with sigma_rule, kql_query, and query_results. Tools include Kusto query/schema helpers, CTI retrieval, MITRE lookup, and Sigma lookup. bash and python run in a per-sample sandbox without Kusto network access. Four tasks: cti_realm_25, cti_realm_50, cti_realm_25_minimal (CTI tools removed), cti_realm_25_seeded (workflow files in /memories/).
No model card in ModelSpec reports this benchmark yet.