Evaluation

All numbers on this page come from committed files in results/, included here verbatim. They were produced by benchmark run 37094883465 (a cold run on a clean ubuntu-24.04 runner) and live-telemetry run 37092465944; every generated file names its run and commit in its first lines.

Methodology

Ground truth. OTRF and Splunk recordings carry ATT&CK labels per recording. A rule fires on-target on a recording when one of its ATT&CK tags is in the same technique family as one of the recording's labels (T1059 / T1059.001 / T1059.003 are one family). Exact-ID coverage requires the same ID. Revoked IDs (for example T1086) are first mapped to their current replacement. The live job labels individual events: every audit record whose SYSCALL PID belongs to an allowlisted command.

Outcomes. For each technique: detected = every recording of it has an on-target alert; partial = some do; missed = none do. Technique coverage counts partial as covered. The stricter fully detected coverage is reported next to it.

Claimed vs measured. Claimed = at least one rule in the set is tagged with the technique's family. This is what a tag-generated ATT&CK Navigator layer reports. Measured = such a rule fired on-target. Claimed-but-missed techniques are split into telemetry gaps (no tagged rule's target (channel, EventID) occurs in any recording of the technique) and rule-logic gaps (events of a tagged rule's target log source and event type are in a recording, but no tagged rule matched). Measured coverage is nested in claimed coverage by construction, so every discordant technique points the same way: an exact McNemar test (for OTRF sigma-all, p = 0.001, exactly 0.00098, from 11 discordant techniques) would only restate their count. The overstatement is therefore reported as claimed-but-not-measured techniques / techniques, in points, with a Wilson interval. The same holds for sigma-core vs sigma-all, channel ablation and leakage control, which can only lose (or only gain) detections. Exact McNemar tests are kept for non-nested comparisons: CTID has_sigma vs measured, and OTRF vs Splunk outcomes.

Off-target alerts. Rules that fire but are not on-target. The recordings contain lab background activity and adjacent attack steps, so this is an upper bound on alert burden, not a false-positive rate.

Threat weighting. Each technique is weighted by its relevance to a profile: the share of the profile's ATT&CK groups that use it.

Rule sets.

• legacy: the 6 replayable v0.1 hand-written rules.
• sigma-core: SigmaHQ release package rules with status stable/test and level high/critical.
• sigma-all: every Windows rule in the SigmaHQ sigma_all_rules release package. That package contains medium, high and critical rules only, and no threat-hunting rules.
• sigma-full: a checkout of the same tag that adds low and informational rules and rules-threat-hunting/. It is used in the cross-dataset and live runs.

Uncertainty. Every bracket says what it is where it appears:

• 95% Wilson for proportions (coverage, recall, overstatement, agreement), computed from the raw counts and rounded once. Wilson treats recordings and techniques as independent; a cluster bootstrap by technique family would be wider.
• 95% percentile bootstrap for threat-weighted coverage (2,000 resamples of techniques, seed 0) and for strategy means over held-out groups.
• 95% paired percentile bootstrap plus an exact sign test for prioritization and prediction differences on the same held-out groups. With fewer than 10 held-out groups (the cloud profile has 4), no CI is printed and the per-group differences are shown instead. Prediction uses leave-one-group-out with 5 hide-split seeds.
• 2.5-97.5 percentile of 200 random draws for the random-rule baseline of the held-out selection.

Descriptive figures (off-target alert burden, the in-sample greedy path) say so and carry no interval.

Leakage control. Rules whose references or description cite OTRF, Security-Datasets, the Threat Hunter Playbook, or the OTRF co-founder's blog and handles are removed. Coverage is then recomputed.

Headline: claimed vs measured coverage

Cross-dataset benchmark: claimed vs measured coverage

Generated by python -m gauntlet extended (SigmaHQ r2026-07-01, ATT&CK v19.2). Source: GitHub Actions run 37094883465 at commit d501d40. Claimed = at least one rule is tagged with the technique (family match); measured = such a rule fired on-target on a recording of it. Technique coverage counts partially detected techniques as covered; fully detected requires every recording of the technique to be detected. Brackets are 95% Wilson intervals computed from the raw counts and rounded once.

Measured is a subset of claimed by construction (a technique can only be measured by a rule tagged with it), so every discordant technique points the same way and a McNemar test would only restate their count. The overstatement is therefore reported as claimed-but-not-measured techniques / techniques, in points, with a Wilson interval. Gaps: a tagged rule's (channel, EventID) target never occurs in a recording of the technique (telemetry) or does occur but no tagged rule matched (rule logic).

Source	Rule set	Recordings	Techniques	Claimed [95% CI]	Measured [95% CI]	Fully detected [95% CI]	Claimed, not measured	Overstatement, pts [95% CI]	Gaps (telemetry / rule logic)
otrf	sigma-all	98	55	100.0% [93.5, 100.0]	80.0% [67.6, 88.4]	58.2% [45.0, 70.3]	11	20.0 [11.6, 32.4]	0 / 11
otrf	sigma-full	98	55	100.0% [93.5, 100.0]	85.5% [73.8, 92.4]	67.3% [54.1, 78.2]	8	14.5 [7.6, 26.2]	0 / 8
otrf-compound	sigma-all	7	13	100.0% [77.2, 100.0]	23.1% [8.2, 50.3]	23.1% [8.2, 50.3]	10	76.9 [49.7, 91.8]	0 / 10
otrf-compound	sigma-full	7	13	100.0% [77.2, 100.0]	61.5% [35.5, 82.3]	53.8% [29.1, 76.8]	5	38.5 [17.7, 64.5]	0 / 5
splunk-windows	sigma-all	414	195	94.9% [90.8, 97.2]	35.9% [29.5, 42.8]	14.9% [10.6, 20.5]	115	59.0 [52.0, 65.6]	22 / 93
splunk-windows	sigma-full	414	195	96.4% [92.8, 98.3]	37.9% [31.4, 44.9]	15.4% [11.0, 21.1]	114	58.5 [51.4, 65.1]	21 / 93
splunk-linux	sigma-all	154	55	70.9% [57.9, 81.2]	10.9% [5.1, 21.8]	1.8% [0.3, 9.6]	33	60.0 [46.8, 71.9]	10 / 23
splunk-linux	sigma-full	154	55	78.2% [65.6, 87.1]	14.5% [7.6, 26.2]	3.6% [1.0, 12.3]	35	63.6 [50.4, 75.1]	9 / 26

Exact-ID variant (a rule tagged with exactly the recorded ID, not just its family):

Source	Rule set	Claimed (exact ID) [95% CI]	Measured (exact ID) [95% CI]	Overstatement, pts [95% CI]
otrf	sigma-all	92.7% [82.7, 97.1]	65.5% [52.3, 76.6]	27.3 [17.3, 40.2]
otrf	sigma-full	92.7% [82.7, 97.1]	74.5% [61.7, 84.2]	18.2 [10.2, 30.3]
otrf-compound	sigma-all	84.6% [57.8, 95.7]	23.1% [8.2, 50.3]	61.5 [35.5, 82.3]
otrf-compound	sigma-full	84.6% [57.8, 95.7]	53.8% [29.1, 76.8]	30.8 [12.7, 57.6]
splunk-windows	sigma-all	81.0% [74.9, 85.9]	29.2% [23.3, 36.0]	51.8 [44.8, 58.7]
splunk-windows	sigma-full	83.6% [77.8, 88.1]	30.8% [24.7, 37.6]	52.8 [45.8, 59.7]
splunk-linux	sigma-all	52.7% [39.8, 65.3]	7.3% [2.9, 17.3]	45.5 [33.0, 58.5]
splunk-linux	sigma-full	65.5% [52.3, 76.6]	10.9% [5.1, 21.8]	54.5 [41.5, 67.0]

Events whose channel could not be parsed (no rule can match them): otrf 0 of 757090 events in 0 recordings; otrf-compound 0 of 354229 events in 0 recordings; splunk-windows 39 of 23037 events in 7 recordings; splunk-linux 4 of 10653 events in 1 recordings.

OTRF vs Splunk (Windows, sigma-all): 38 techniques recorded in both; measured outcome agrees on 31 of them (81.6%, 95% Wilson [66.6, 90.8]); exact McNemar p = 0.125 for the 6 vs 1 discordant techniques. Detected only on OTRF: T1003.003, T1003.006, T1055.001, T1134.001, T1218, T1218.010; only on Splunk: T1059.001.

Held-out rule selection: pick 10 rules on OTRF, test on Splunk Windows

Weighted coverage brackets are 95% percentile bootstrap intervals over test techniques. The last column is the share of random draws whose test ransomware-weighted coverage is at or above the selection's (a one-sided permutation p-value).

Selection on OTRF	Test technique coverage	Test ransomware-weighted coverage	Random draws at or above
CTI-weighted greedy	0.5% (1 of 195, 95% Wilson [0.1, 2.8])	0.8% [0.0, 2.5]	69.0%
Unweighted greedy	1.0% (2 of 195, 95% Wilson [0.3, 3.7])	1.8% [0.0, 4.5]	24.5%
Random 10 of 111 OTRF-firing rules (200 draws, mean [2.5-97.5 percentile of the draws])	0.9% [0.0%, 2.1%]	1.2% [0.0%, 3.2%]	-

Recordings skipped because a file was missing or unreadable: otrf: SDWIN-230718150800.

Runtime 124.7 s.

Control: is the gap the rules or the evaluator?

The claimed-vs-measured gap could partly be an artefact of GAUNTLET's in-house Sigma evaluator. SigmaHQ ships positive regression samples for some rules; replaying each sample through the rule it was recorded for isolates evaluator fidelity from rule logic.

Evaluator self-test on SigmaHQ regression data

Generated by python -m gauntlet selftest on the SigmaHQ r2026-07-01 checkout. Source: GitHub Actions run 37094883465 at commit d501d40.

Each SigmaHQ positive regression sample is replayed through the one rule it was recorded for. 197 of 197 supported rules fire on their own sample: recall 100.0% (95% Wilson [98.1, 100.0]). 163 of 197 also match exactly the expected number of events. A rule GAUNTLET's evaluator cannot parse is not tested (see below); this checks evaluator fidelity on the rules it does run, which is what the claimed-vs-measured gap depends on.

Sample category	Tested	Fired
windows/process_creation	128	128
windows/registry	32	32
windows/file	18	18
windows/image_load	4	4
2025/Malware	4	4
2026/Exploits	4	4
windows/builtin	3	3
windows/process_access	2	2
windows/sysmon	1	1
2025/Exploits	1	1

Samples not tested: none.

Live auditd telemetry vs replayed recordings

The live-telemetry workflow runs on an ephemeral ubuntu-24.04 runner. It runs allowlisted, read-only discovery commands under auditd (ADR 0005), then scores the Linux SigmaHQ rules on that telemetry, event by event.

Four runner executions (runs 36995051317, 36998137511, 37016216096 and 37092465944) gave identical per-command outcomes (results/live-history.json). Three findings stand out:

• The release package's Linux rules detect none of the commands. The rules that would match (user, system and process discovery) are low or informational level, or threat-hunting rules, and the package does not ship those.
• None of the 12 Splunk Linux recordings of the same techniques is detected. 8 are auditd extracts with no EXECVE record, so no command line exists for a process-creation rule to match. The other 4 are Sysmon for Linux recordings that do contain process-creation events (id, crontab -l); the rules that fire on them are tagged with other techniques (T1087.001, T1007). The live-vs-replay difference is therefore partly missing telemetry and partly label/tag disagreement, not evidence that replay generally underestimates live coverage.
• crontab -l is tagged T1007 (System Service Discovery) upstream, while the live job and Splunk label it T1053.003 (Scheduled Task/Job: Cron), so it counts as off-target: a taxonomy disagreement, not a detection miss.

Live auditd telemetry vs replayed recordings

Generated by python -m gauntlet live in the live-telemetry workflow. Source: GitHub Actions run 37092465944 at commit 82694b3.

Rule set: sigma-all (128 Linux rules)

256 audit records -> 281 events (67 labelled to allowlisted commands, 214 background). Live: 0 of 4 techniques detected (n = 4, 95% Wilson [0.0, 49.0]; claimed by tags: 3 of 4). Rules firing on background: 1.

Command	Technique	Labelled events	Claimed	Live: detected	Replayed Splunk auditd: detected	Replayed Splunk Sysmon for Linux: detected
whoami	T1033	7	yes	no	0 of 2	0 of 2
/usr/bin/whoami	T1033	7	yes	no	0 of 2	0 of 2
id	T1033	7	yes	no	0 of 2	0 of 2
uname -a	T1082	7	yes	no	0 of 2	0 of 1
hostname	T1082	7	yes	no	0 of 2	0 of 1
cat /etc/os-release	T1082	11	yes	no	0 of 2	0 of 1
ps -ef	T1057	7	no	no	n/a (no recording)	n/a (no recording)
crontab -l	T1053.003	7	yes	no	0 of 4	0 of 1
ls -la /etc/cron.d	T1053.003	7	yes	no	0 of 4	0 of 1

Rule set: sigma-full (192 Linux rules)

256 audit records -> 281 events (67 labelled to allowlisted commands, 214 background). Live: 3 of 4 techniques detected (n = 4, 95% Wilson [30.1, 95.4]; claimed by tags: 4 of 4). Rules firing on background: 1.

Command	Technique	Labelled events	Claimed	Live: detected	Replayed Splunk auditd: detected	Replayed Splunk Sysmon for Linux: detected
whoami	T1033	7	yes	yes	0 of 2	0 of 2
/usr/bin/whoami	T1033	7	yes	no	0 of 2	0 of 2
id	T1033	7	yes	yes	0 of 2	0 of 2
uname -a	T1082	7	yes	yes	0 of 2	0 of 1
hostname	T1082	7	yes	yes	0 of 2	0 of 1
cat /etc/os-release	T1082	11	yes	yes	0 of 2	0 of 1
ps -ef	T1057	7	yes	yes	n/a (no recording)	n/a (no recording)
crontab -l	T1053.003	7	yes	no	0 of 4	0 of 1
ls -la /etc/cron.d	T1053.003	7	yes	no	0 of 4	0 of 1

Splunk EXECVE completeness: why the replayed recordings miss

12 Splunk Linux recordings are labelled with these techniques. 8 are auditd extracts; 0 of them contain an EXECVE record, so no command line exists for a process_creation rule to match. 4 are Sysmon for Linux recordings; 4 of them contain process-creation events (EventID 1). Recordings detected by a rule tagged with the recorded technique: sigma-all 0 of 12, sigma-full 0 of 12. Where other rules fired, they are tagged with a different technique (listed below), so the miss on those recordings is a labelling or tagging difference, not missing telemetry.

Recording	Technique	Sourcetype	Events	EXECVE	SYSCALL	Sysmon EventID 1	sigma-all: detected	sigma-full: detected	Other rules fired (sigma-full if present)
SPLK-4cb6c050-1ab2-11f0-a1c6-629be3538069	T1033	auditd	3	0	3	0	no	no	-
SPLK-77d03c0c-5e07-11ef-b158-acde48001122	T1033	auditd	3	0	3	0	no	no	-
SPLK-1ea06ecf-1423-4f00-a3b0-66bc2be38008@linux	T1033	sysmon:linux	2	0	0	1	no	no	Local System Accounts Discovery - Linux (T1087.001)
SPLK-d01afb53-3320-46ce-9590-31df1f5c11b4	T1033	sysmon:linux	4	0	0	2	no	no	Local System Accounts Discovery - Linux (T1087.001)
SPLK-0a6dfea0-5647-11ef-b567-acde48001122	T1053.003	auditd	3	0	1	0	no	no	-
SPLK-4bed5528-1ab0-11f0-a1c6-629be3538069	T1053.003	auditd	3	0	3	0	no	no	-
SPLK-6f8a621e-45d5-11f0-9bec-629be3538068	T1053.003	auditd	19	0	0	0	no	no	-
SPLK-ebb93346-cab0-11f0-9d54-629be353806a	T1053.003	auditd	20	0	0	0	no	no	-
SPLK-3c9236a6-c537-11ec-bdfa-acde48001122	T1053.003	sysmon:linux	332	0	0	44	no	no	Crontab Enumeration (T1007); File Deletion (T1070.004); Process Discovery (T1057)
SPLK-10da084a-1ab1-11f0-a1c6-629be3538069	T1082	auditd	1	0	1	0	no	no	-
SPLK-f0ce5c20-5645-11ef-b567-acde48001122	T1082	auditd	1	0	1	0	no	no	-
SPLK-678e0d41-1fb6-40e4-8556-fc5168784634@linux	T1082	sysmon:linux	1	0	0	1	no	no	-

Comparison with published numbers

Comparison with published coverage numbers

Generated by python -m gauntlet compare. Source: GitHub Actions run 37094883465 at commit d501d40. Inputs: live run 37092465944, extended run 37094883465.

CTID Top ATT&CK Techniques: has_sigma (claimed) vs measured on OTRF (sigma-all)

Of the 53 OTRF-recorded techniques listed by CTID, CTID flags 50 (94.3%, 95% Wilson [84.6, 98.1]) as having a Sigma rule; GAUNTLET measures an on-target detection for 42 (79.2% [66.5, 88.0]) (exact McNemar p = 0.0574, 11 vs 3 discordant techniques; the two are not nested, so the test applies). Claimed but not measured: T1012, T1049, T1056.002, T1059.001, T1069.002, T1087.001, T1087.002, T1110.003, T1135, T1210, T1222.001. Measured but not flagged by CTID: T1055.002, T1218.004, T1558.003.

Not like-for-like: CTID's flag dates from its own Sigma snapshot and says a rule exists; GAUNTLET pins SigmaHQ r2026-07-01 and requires the rule to fire on recorded telemetry.

RedGap (benign Linux lab, SigmaHQ linux/process_creation) vs GAUNTLET live auditd and Splunk replay

RedGap's docs/benchmarks/coverage.json at commit a9bcbf2c reports 33/51 techniques detected (its README headline may cite a later run). Techniques both projects exercise:

Technique	RedGap detected (gap type)	GAUNTLET live	GAUNTLET Splunk replay (sigma-full)
T1003.008 OS Credential Dumping: /etc/passwd and /etc/shadow	no (rule)	n/a	no
T1016 System Network Configuration Discovery	yes (none)	n/a	no
T1027 Obfuscated Files or Information	yes (none)	n/a	no
T1033 System Owner/User Discovery	no (rule)	sigma-all: no; sigma-full: yes	no
T1036 Masquerading	yes (none)	n/a	no
T1053.002 Scheduled Task/Job: At	yes (none)	n/a	no
T1053.003 Scheduled Task/Job: Cron	yes (none)	sigma-all: no; sigma-full: no	no
T1057 Process Discovery	no (base_rate)	sigma-all: no; sigma-full: yes	n/a
T1059 Command and Scripting Interpreter	yes (none)	n/a	no
T1059.004 Command and Scripting Interpreter: Unix Shell	yes (none)	n/a	yes
T1082 System Information Discovery	yes (none)	sigma-all: no; sigma-full: yes	no
T1083 File and Directory Discovery	yes (none)	n/a	no
T1098.004 Account Manipulation: SSH Authorized Keys	no (rule)	n/a	no
T1105 Ingress Tool Transfer	yes (none)	n/a	no
T1136.001 Create Account: Local Account	no (rule)	n/a	yes
T1140 Deobfuscate/Decode Files or Information	yes (none)	n/a	no
T1222.002 Linux and Mac File and Directory Permissions Modification	yes (none)	n/a	no
T1485 Data Destruction	yes (none)	n/a	no
T1489 Service Stop	yes (none)	n/a	no
T1543.002 Create or Modify System Process: Systemd Service	no (rule)	n/a	no
T1546.004 Event Triggered Execution: Unix Shell Configuration Modification	no (base_rate)	n/a	no
T1548 Abuse Elevation Control Mechanism	yes (none)	n/a	no
T1548.001 Abuse Elevation Control Mechanism: Setuid and Setgid	yes (none)	n/a	no
T1552.004 Unsecured Credentials: Private Keys	no (rule)	n/a	yes
T1567 Exfiltration Over Web Service	yes (none)	n/a	no

Not like-for-like: RedGap uses its own fixture telemetry mapped to Sysmon-for-Linux process events and a different SigmaHQ commit; GAUNTLET's live job uses raw auditd. No published technique-coverage figure exists for SigmaHQ on OTRF or Splunk attack_data recordings, and MITRE ATT&CK Evaluations score vendor products, not open rule sets, so they are not compared.

Full benchmark tables (OTRF atomic)

GAUNTLET benchmark results

ATT&CK Enterprise v19.2 - SigmaHQ r2026-07-01 - 98 OTRF Security-Datasets Windows recordings covering 55 techniques. Generated by python -m gauntlet bench; skipped (file missing or unreadable): SDWIN-230718150800. Source: GitHub Actions run 37094883465 at commit d501d40.

Detection coverage on real recorded attack telemetry

Brackets are 95% Wilson intervals, computed from the raw counts and rounded once (55 techniques / 98 recordings are small samples; recordings are treated as independent). Technique coverage = at least one recording of the technique has an on-target alert (partially detected techniques count); fully detected = every recording of it does. The off-target columns are descriptive (an upper bound on alert burden in one lab; alerts cluster within recordings, so no interval is given).

Rule set	Rules	Technique coverage	Fully detected	Exact-ID coverage	Coverage w/o OTRF-citing rules	Recordings detected	Off-target rules / recording	Off-target alerts / 10k events
legacy	6	5.5% [1.9, 14.9]	1.8% [0.3, 9.6]	3.6% [1.0, 12.3]	n/a	3.1% [1.0, 8.6]	0.14	0.22
sigma-core	1365	65.5% [52.3, 76.6]	40.0% [28.1, 53.2]	56.4% [43.3, 68.6]	58.2% [45.0, 70.3] (-25 rules)	52.0% [42.3, 61.7]	1.52	11.13
sigma-all	2519	80.0% [67.6, 88.4]	58.2% [45.0, 70.3]	65.5% [52.3, 76.6]	76.4% [63.7, 85.6] (-50 rules)	68.4% [58.6, 76.7]	3.43	53.53

sigma-core to sigma-all: 8 of 55 techniques gained (+14.5 points, 95% Wilson [7.6, 26.2]), 0 lost. sigma-core is a filter of sigma-all, so no technique can be lost and a McNemar test would only restate the count; the interval on the gain is reported instead.

Leakage control removes rules whose references or description cite OTRF / Security-Datasets / Threat Hunter Playbook or the OTRF co-founder's blog and handles. For sigma-core it loses 4 of 55 techniques (-7.3 points, 95% Wilson [2.9, 17.3]). With the narrower v1.0 pattern (OTRF names only) sigma-core coverage would be 60.0%.

Threat-weighted coverage per CTI profile

Each technique is weighted by the share of the profile's groups that use it. Brackets are 95% percentile bootstrap intervals (2,000 resamples of the recorded techniques, seed 0).

Profile	ATT&CK groups	legacy	sigma-core	sigma-all
ransomware	18	7.5% [0.0, 16.7]	66.5% [52.8, 79.4]	79.7% [66.9, 90.8]
espionage	57	8.1% [0.0, 18.1]	66.1% [50.8, 79.9]	80.0% [66.0, 91.2]
financial	24	8.5% [0.0, 18.7]	66.7% [52.4, 79.6]	80.9% [67.6, 91.4]
cloud	4	6.4% [0.0, 15.0]	68.1% [54.0, 80.7]	83.0% [71.6, 92.2]

Coverage sprint (demo scenario 5)

Hand-written baseline rules: 5.5% (3 of 55) [1.9, 14.9] technique coverage (7.5% [0.0, 16.7] ransomware-weighted). Adding only the top-10 greedy cheapest-win rules from sigma-all (listed below): 34.5% (19 of 55) [23.4, 47.7] (43.8% [30.3, 58.6] ransomware-weighted). Wilson and bootstrap intervals as above. In-sample: the rules are selected and scored on the same recordings, so this is optimistic and the intervals do not cover the selection; the held-out OTRF-to-Splunk check is in EXTENDED.md.

Cheapest wins (greedy, ransomware-weighted, sigma-all (used by the sprint))

The cumulative column is the in-sample greedy path (descriptive, no interval).

#	Rule	New techniques	Cumulative weighted coverage
1	Remote Thread Creation In Uncommon Target Image	T1055, T1055.001, T1055.002, T1055.003	8.8%
2	CobaltStrike Service Installations - Security	T1021, T1021.002	14.1%
3	LSASS Access From Non System Account	T1003.001, T1003.002	19.2%
4	Unsigned DLL Loaded by Windows Utility	T1218.002, T1218.004	23.3%
5	Legitimate Application Writing Files In Uncommon Location	T1218.001, T1218.005	27.4%
6	PCRE.NET Package Image Load	T1059	30.6%
7	DPAPI Domain Backup Key Extraction	T1003	33.6%
8	T1047 Wmiprvse Wbemcomn DLL Hijack	T1047	36.5%
9	Meterpreter or Cobalt Strike Getsystem Service Installation - Security	T1134.001, T1134.002	39.3%
10	Dumping of Sensitive Hives Via Reg.EXE	T1003.004	41.7%

Cheapest wins (greedy, ransomware-weighted, sigma-core)

The cumulative column is the in-sample greedy path (descriptive, no interval).

#	Rule	New techniques	Cumulative weighted coverage
1	CobaltStrike Service Installations - Security	T1021, T1021.002	5.3%
2	Dumping of Sensitive Hives Via Reg.EXE	T1003.002, T1003.004	10.2%
3	PCRE.NET Package Image Load	T1059	13.3%
4	DPAPI Domain Backup Key Extraction	T1003	16.4%
5	T1047 Wmiprvse Wbemcomn DLL Hijack	T1047	19.2%
6	Meterpreter or Cobalt Strike Getsystem Service Installation - Security	T1134.001, T1134.002	22.0%
7	LSASS Dump Keyword In CommandLine	T1003.001	24.6%
8	NetNTLM Downgrade Attack - Registry	T1112	27.1%
9	Successful Overpass the Hash Attempt	T1550.002	29.5%
10	Suspicious Service Path Modification	T1543.003	32.0%

Telemetry ablation (sigma-core): techniques lost if a channel is not collected

Dropping a channel can only lose detections, so the loss is reported as points of technique coverage with a 95% Wilson interval on lost / recorded techniques.

Channel	Techniques lost	Coverage lost, points [95% Wilson]	Coverage without
microsoft-windows-sysmon/operational	5	9.1 [3.9, 19.6]	56.4%
security	3	5.5 [1.9, 14.9]	60.0%
microsoft-windows-powershell/operational	1	1.8 [0.3, 9.6]	63.6%
windows powershell	0	0.0 [0.0, 6.5]	65.5%
system	0	0.0 [0.0, 6.5]	65.5%

Per-tactic coverage (sigma-core)

Tactic	Covered / recorded techniques	95% Wilson (%)
collection	0/1	[0.0, 79.3]
credential-access	6/9	[35.4, 87.9]
defense-impairment	1/3	[6.1, 79.2]
discovery	1/8	[2.2, 47.1]
execution	6/7	[48.7, 97.4]
lateral-movement	5/6	[43.6, 97.0]
persistence	5/6	[43.6, 97.0]
privilege-escalation	5/7	[35.9, 91.8]
stealth	7/8	[52.9, 97.8]

Prioritization: CTI-ranked vs breadth-first emulation (leave-one-group-out)

Recall of a held-out actor's techniques after k emulations, universe = Windows techniques with an Atomic Red Team test. Lower steps to 80% is better.

Profile	Strategy	recall@10	recall@25	recall@50	steps to 50%	steps to 80% [95% CI]	AUC [95% CI]
ransomware (n=18)	cti	0.070	0.201	0.376	64.3	118.9 [114.3, 123.3]	0.718 [0.707, 0.728]
ransomware (n=18)	relevance	0.066	0.200	0.418	62.3	126.6 [119.4, 133.6]	0.710 [0.696, 0.722]
ransomware (n=18)	prevalence	0.080	0.195	0.381	66.8	126.9 [120.6, 132.9]	0.710 [0.698, 0.722]
ransomware (n=18)	breadth	0.061	0.140	0.255	99.8	194.4 [185.8, 202.0]	0.568 [0.555, 0.582]
ransomware (n=18)	random	0.041	0.095	0.186	131.5	211.9 [210.5, 213.1]	0.502 [0.500, 0.504]
espionage (n=57)	cti	0.114	0.309	0.516	52.2	103.2 [95.2, 111.0]	0.761 [0.744, 0.778]
espionage (n=57)	relevance	0.109	0.305	0.515	53.2	106.0 [97.3, 114.2]	0.755 [0.737, 0.773]
espionage (n=57)	prevalence	0.127	0.303	0.511	52.1	103.2 [95.2, 111.0]	0.761 [0.744, 0.778]
espionage (n=57)	breadth	0.045	0.125	0.244	93.5	196.8 [189.7, 203.0]	0.580 [0.569, 0.591]
espionage (n=57)	random	0.042	0.096	0.189	131.5	211.2 [209.3, 212.6]	0.503 [0.501, 0.505]
financial (n=24)	cti	0.080	0.214	0.443	59.5	123.7 [117.6, 129.9]	0.727 [0.714, 0.739]
financial (n=24)	relevance	0.068	0.213	0.427	60.2	134.8 [126.7, 142.8]	0.710 [0.697, 0.723]
financial (n=24)	prevalence	0.091	0.239	0.436	58.8	124.5 [117.6, 131.2]	0.727 [0.714, 0.738]
financial (n=24)	breadth	0.041	0.117	0.236	95.5	201.9 [193.5, 209.8]	0.569 [0.557, 0.580]
financial (n=24)	random	0.042	0.096	0.188	131.8	211.8 [210.0, 213.1]	0.502 [0.500, 0.505]
cloud (n=4)	cti	0.109	0.238	0.448	61.2	147.2	0.691
cloud (n=4)	relevance	0.090	0.222	0.405	73.0	178.8	0.640
cloud (n=4)	prevalence	0.118	0.253	0.437	61.8	123.0	0.728
cloud (n=4)	breadth	0.062	0.129	0.265	94.5	196.2	0.577
cloud (n=4)	random	0.040	0.095	0.184	130.8	210.7	0.502

Brackets: 95% percentile bootstrap over held-out groups (2,000 resamples, seed 0); not shown for profiles with fewer than 10 groups.

Paired comparison over held-out groups (mean difference, 95% paired percentile bootstrap CI and exact sign-test p; negative steps = the first strategy needs fewer emulations). With fewer than 10 held-out groups the bootstrap CI is not reported; the per-group differences are shown instead.

Profile	Comparison	Δ steps to 80%	sign p	Δ AUC
ransomware (n=18)	cti minus breadth	-75.5 [-85.0, -65.8]	<0.0001	+0.150 [+0.136, +0.162]
ransomware (n=18)	cti minus prevalence	-8.0 [-12.2, -3.9]	0.000145	+0.008 [+0.003, +0.013]
ransomware (n=18)	cti minus random	-93.0 [-97.8, -88.4]	<0.0001	+0.215 [+0.205, +0.226]
ransomware (n=18)	breadth minus random	-17.5 [-25.6, -10.5]	0.00754	+0.066 [+0.053, +0.079]
espionage (n=57)	cti minus breadth	-93.6 [-102.2, -85.0]	<0.0001	+0.181 [+0.165, +0.199]
espionage (n=57)	cti minus prevalence	+0.0 [-1.0, +0.9]	0.471	+0.000 [-0.001, +0.001]
espionage (n=57)	cti minus random	-108.0 [-115.8, -100.6]	<0.0001	+0.258 [+0.240, +0.276]
espionage (n=57)	breadth minus random	-14.4 [-21.4, -8.0]	<0.0001	+0.077 [+0.066, +0.088]
financial (n=24)	cti minus breadth	-78.2 [-88.3, -68.4]	<0.0001	+0.158 [+0.143, +0.174]
financial (n=24)	cti minus prevalence	-0.8 [-3.6, +1.9]	0.678	-0.000 [-0.002, +0.002]
financial (n=24)	cti minus random	-88.1 [-93.8, -82.0]	<0.0001	+0.224 [+0.212, +0.236]
financial (n=24)	breadth minus random	-9.9 [-18.2, -1.2]	0.00661	+0.066 [+0.054, +0.078]
cloud (n=4)	cti minus breadth	-49.0 (per group: -9, -38, -84, -65)	0.125	+0.114
cloud (n=4)	cti minus prevalence	+24.2 (per group: +56, +7, +11, +23)	0.125	-0.036
cloud (n=4)	cti minus random	-63.4 (per group: -10, -70, -103, -71)	0.125	+0.190
cloud (n=4)	breadth minus random	-14.4 (per group: -1, -32, -19, -6)	0.125	+0.075

Next-technique prediction (co-occurrence vs popularity, leave-one-group-out)

Level	Model	recall@5	recall@10 [95% CI]	recall@20	MRR [95% CI]
sub_technique_level (n=165)	cooccurrence	0.131	0.235 [0.215, 0.257]	0.366	0.897 [0.857, 0.934]
sub_technique_level (n=165)	popularity	0.114	0.191 [0.174, 0.209]	0.311	0.865 [0.820, 0.908]
technique_level (n=168)	cooccurrence	0.172	0.311 [0.293, 0.332]	0.495	0.909 [0.875, 0.941]
technique_level (n=168)	popularity	0.168	0.278 [0.261, 0.297]	0.456	0.926 [0.891, 0.958]

Brackets: 95% percentile bootstrap over held-out groups (seed 0 hide split).

Level	co-occurrence − popularity recall@10 (95% paired bootstrap CI)	co-occurrence − popularity MRR (95% paired bootstrap CI)	recall@10 mean ± sd over 5 hide-split seeds
sub_technique_level	+0.044 [+0.026, +0.064]	+0.032 [-0.018, +0.079]	cooccurrence 0.237 ± 0.003; popularity 0.194 ± 0.001
technique_level	+0.034 [+0.021, +0.048]	-0.017 [-0.054, +0.021]	cooccurrence 0.311 ± 0.005; popularity 0.277 ± 0.002

Sigma evaluator support

2519 Windows rules evaluated, 35 skipped as unsupported. Top reasons: service appxdeployment-server (9), service msexchange-management (8), category file_access (6), service iis-configuration (3), service dns-server-analytic (1), service smbclient-connectivity (1), service appxpackaging-om (1), service certificateservicesclient-lifecycle-system (1).

Runtime: detection 70.2 s, prioritization 2.7 s, prediction 14.1 s; total 87.0 s.