Check whether SIEM detection rules fire by running Atomic Red Team tests

Kelvin Santos wanted to know whether the detection rules in his detection-as-code repo really catch the ATT&CK techniques they are mapped to. He built a custom agent with LangChain and gpt-4o-mini, connected to the community Atomic Red Team MCP server. Given a rule, the model searches the existing atomic tests and names the one that matches the technique; it is only allowed to query and run existing atomics, never to generate new attack code. The test runs on an isolated Windows VM built with Terraform, with Sysmon logs shipped to Microsoft Sentinel. Code then asks Sentinel two things for the run window: did any process telemetry arrive at all, and did the rule's own query match. The result is detected, gap (data arrived, rule missed it) or blind (no data at all).

On a PowerShell encoded-command rule, the first run came back as a gap: the agent picked an atomic that passed encoded arguments with -Command, Sysmon logged it, and the rule stayed quiet because it only looks for -EncodedCommand style flags. A second run picked a different atomic that used -EncodedCommand, and the rule fired, with the scheduled Sentinel rule later raising a real incident. On a gap or blind result the agent opens an issue in the rules repo with the rule, technique, test, time window and query, labelled by type, and skips duplicates by title.

Each run samples one atomic, so one result says little about the whole technique; the same rule flipped between gap and detected. Techniques without an atomic test cannot be checked this way. The atomics do real and sometimes destructive things, so keep the VM isolated and disposable. The author keeps the decision about what a gap means with a person. The code is on GitHub.