Skip to content

Limitations and roadmap

Limitations

  • Engine scope. expand placeholders, aggregations and correlations are not evaluated. That affects 2 of the 3,757 rules; the 23-rule rules-placeholder folder is excluded up front, and SigmaHQ's legacy aggregations live in unsupported/ (counts in the Results). Linux and AWS routing is new in 1.1.0 (ADR 0007). Azure/M365 is not routed.
  • Benign corpus. The three hosts are clean lab installs recorded from a fresh install (Sysmon and audit policy, then software installation, through Ninite on the Win10 host, and basic user interaction). Per-day rates of install-related rules (scheduled-task registry writes, AppX installs, AppCompat) are therefore likely inflated relative to steady state, while a production fleet's own noise sources are missing, so neither the counts nor the rates bound a real fleet; only the set of rules that fire is plausibly a lower bound. Rates are per host-day. There is no benign CloudTrail corpus, and the benign Linux sample recorded on the CI runner is a short developer workload: a parser smoke test, not an FP rate.
  • Coarse labels. OTRF, Splunk and the runner's own emulation window are labelled per technique and include background noise. "Technique detected" means a rule tagged with the technique or its parent fired.
  • Drafter evaluation is optimistic by construction, because drafts are tested on the dataset they were drafted from.
  • Decay. The inventory is a fleet union, which hides changes that reach only some hosts (the mixed row). In the simulated rows the same transform builds the ground truth and the post-change inventory, so their 100% precision is partly a soundness property of the transform; the real OTRF check avoids that but covers one change (Sysmon removal) and labels rules by whether they fire at all, not by technique.
  • History. Two synthetic telemetry files over 1 MB remain in early git history (removed in c8cf57f); history is not rewritten.
  • Timings come from shared machines; treat throughput as indicative.
  • Deferred, with reasons.
  • LLM drafter benchmark: the backend is implemented, but no API key is available to this project, and a fair benchmark also needs blind human review of the drafts.
  • Benign CloudTrail corpus: no public, attack-free CloudTrail corpus was found; the public CloudTrail sets are attack simulations.
  • GAUNTLET integration: waits on a stable GAUNTLET capture export. Both projects already replay the same 98 OTRF recordings, compared in the table above.
  • Per-host field inventories: three benign hosts with different roles are too few to validate a per-host model.
  • Azure/M365 routing: no public labelled Azure/M365 corpus that can be pinned and checksummed, so routing could not be validated.
  • Real inventory for the weekly decay job: .github/workflows/decay.yml runs weekly and opens an issue, but on the bundled synthetic telemetry, because the project has no maintained production inventory.
  • Per-rule time budget for |re: lint warns on ReDoS-prone patterns (W213); the engine uses stdlib re without a timeout.

Roadmap

Each item is deferred for the reason given under Limitations.

  • Point the weekly decay job at a real telemetry export (needs a maintained inventory source)
  • Per-host field inventories for partial rollouts (needs more than three benign hosts to validate)
  • Benchmark the LLM drafter with blind human review (needs an API key and reviewers)
  • Benign CloudTrail corpus; Azure/M365 routing (no public clean or labelled corpus found)
  • GAUNTLET integration: pull emulation captures per technique and push validated coverage back (waits on a stable export)
  • A per-rule time budget for |re evaluation