Skip to content

ANVIL

ANVIL predicts from benign telemetry alone which Sigma detections a telemetry change will silently break: when Sysmon is removed from real OTRF attack captures that also log Security 4688, its per-log-source satisfiability check flags 114 of the 122 rules that stop firing with no false alarm, where field-presence (schema-style) validation raises 63.

Around that core it is an open, SIEM-free CI for Sigma: every rule is linted, replayed on real attack captures and on real clean-host telemetry, gated on a SOC alert budget, and cross-checked against a real OpenSearch backend.

Detection health dashboard

Headline (real data, bench run 37093721154 at 9906b42; k/n, p% [Wilson 95% CI]) Result
Decay monitor, real: Sysmon dropped from 88 OTRF captures that also log native 4688 114/122 rules that stop firing flagged statically, 93.4% [87.6, 96.6]; precision 114/114, 100% [96.7, 100]. Field presence: precision 119/182, 65.4% [58.2, 71.9], 63 false alarms (exact McNemar p = 2.2e-19)
Decay monitor, simulated Sysmon-to-4688 migration on SigmaHQ regression captures 109/115, 94.8% [89.1, 97.6]; precision 109/109, 100% [96.6, 100] (partly by construction: the same transform builds ground truth and inventory). Field presence: 114/272, 41.9% [36.2, 47.8]
TP replay on SigmaHQ regression captures 463/463, 100% [99.2, 100]; 0.1 baseline 420/463, 90.7% [87.7, 93.0]; pySigma -> SQLite 461/463, 99.6% [98.4, 99.9]
Benign replay: 2,994,137 events from 3 clean Windows hosts 75/2,789 observable rules fire, 2.7% [2.2, 3.4]; 12 exceed 20 alerts per host-day (41 at 10 hosts, all 75 at 100)
OTRF Windows emulations technique detected 62/98, 63.3% [53.4, 72.1]; any alert 95/98, 96.9% [91.4, 99.0]
Linux and AWS CloudTrail captures (Splunk attack_data + OTRF) technique detected 27/168, 16.1% [11.3, 22.4]; any alert 89/168, 53.0% [45.4, 60.4]
OpenSearch 2.19 as an independent backend regression verdicts agree 450/458, 98.3% [96.6, 99.1]

How it works Evaluation Reproduce Dashboard GitHub

Try it in 60 seconds

git clone --depth 1 https://github.com/rakshit-737/anvil-detection-engineering && cd anvil
python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -e .                                   # core needs only PyYAML
anvil synth                                        # synthetic benign telemetry
anvil test                                         # -> test: 5/5 rules passed the gate
anvil test --rules examples/noisy                  # -> FAIL ... 719 alerts/day: the SOC budget blocks it
anvil draft examples/cti/fin_x_report.md           # -> 4 drafted rules in drafts/, reviewed: false
anvil lint --rules drafts --profile sigma          # -> lint: 4 rule(s), 4 finding(s), FAIL (A114 review gate)

Or with the container image (non-root; rules and examples bundled):

docker run --rm ghcr.io/rakshit-737/anvil-detection-engineering:v1.1.1 lint --rules rules          # or :latest
docker run --rm -v "$PWD:/work" -w /work ghcr.io/rakshit-737/anvil-detection-engineering:v1.1.1 lint --rules my-rules

Development: pip install -e ".[dev,sigma,ml]" && python -m pytest -q (the real-data tests skip without $ANVIL_DATA).