CLI and API reference

CLI

python -m gauntlet profiles                         # ATT&CK groups per threat profile
python -m gauntlet plan --profile ransomware --top 15
python -m gauntlet manifest --profile ransomware --top 10 --out plan.json   # DRY RUN
python -m gauntlet predict --observed T1566.001,T1059.001 -k 10
python -m gauntlet replay --profile ransomware --top 15 --ruleset sigma-core --navigator layer.json --json report.json [--baseline old.json]   # exit 2 on regression
python -m gauntlet bench [--no-cache]               # OTRF benchmark -> results/
python -m gauntlet extended --full-rules DIR        # cross-dataset, claimed vs measured -> results/extended.json
python -m gauntlet live --audit audit.log --labels labels.json   # score live auditd telemetry (CI job)
python -m gauntlet compare [--live live.json]       # published-number comparison -> results/PUBLISHED.md
python -m gauntlet selftest --sigma-checkout DIR    # evaluator recall on SigmaHQ regression samples -> results/SELFTEST.md
python -m gauntlet kb                               # rebuild the shipped ATT&CK KB
python -m gauntlet sim --profile ransomware         # offline simulation demo

Run python -m gauntlet <command> --help for all options.

Python API

gauntlet.prioritize

CTI prioritizer on real ATT&CK data.

Threat profiles are sets of real ATT&CK groups. By default a profile is selected reproducibly from group descriptions (e.g. every group whose ATT&CK description mentions ransomware), or explicitly by group ids/names.

relevance(T)  = share of profile groups with a documented procedure for T
prevalence(T) = share of *all* ATT&CK groups with a documented procedure for T
score(T)      = relevance(T) * prevalence(T) / max(prevalence)     ("cti")

Alternative strategies (for the research question "does CTI-prioritized emulation reach relevant coverage faster than breadth-first?"): breadth (ATT&CK id order), random, prevalence (global only), relevance (profile only), cti (product, default).

PROFILE_PATTERNS = {'ransomware': ('ransomware', 'Groups whose ATT&CK description mentions ransomware'), 'espionage': ('espionage|intelligence[- ]gathering|intelligence collection', 'Cyber-espionage groups (description mentions espionage / intelligence collection)'), 'financial': ('financially[- ]motivated|financial gain|financial crime', 'Financially-motivated crime groups'), 'cloud': ('\\bcloud\\b|\\bSaaS\\b|Microsoft 365|Office 365|Azure|\\bAWS\\b', 'Groups documented operating against cloud / SaaS environments')} module-attribute

SMALL_N = 10 module-attribute

STRATEGIES = ('cti', 'relevance', 'prevalence', 'breadth', 'random') module-attribute

Ranked dataclass

A technique with its priority score and the components that produced it.

prevalence instance-attribute

relevance instance-attribute

score instance-attribute

technique instance-attribute

__init__(technique, score, relevance, prevalence)

evaluate_logo(kb, profile, universe, ks=(10, 25, 50), random_seeds=30, min_target=5, curve_len=300)

Leave-one-group-out evaluation of prioritization strategies.

For each group g in the profile: relevance is computed from the other profile groups, prevalence from all groups except g, the candidate list is universe (techniques GAUNTLET can emulate), and we measure how quickly each ordering covers g's techniques inside the universe.

prevalence(groups)

Global prevalence: :func:relevance computed over all (or all-but-held-out) ATT&CK groups.

profile_groups(kb, profile)

Resolve a named profile or a comma list of group ids / names / aliases.

rank(candidates, rel, prev, strategy='cti', seed=0)

Order candidate technique ids by a strategy (ties broken by id, deterministic).

recall_curve(order, target)

Fraction of target covered after each step of order (one value per step).

relevance(groups)

Share of groups that use each technique (sub-techniques also credit their parent).

steps_to(curve, level)

First step (1-based) at which curve reaches level; len(curve) + 1 if never.

gauntlet.coverage

Coverage scoring on replayed real telemetry, gap ranking and ATT&CK Navigator export.

Ground truth is the dataset's ATT&CK mapping. A rule firing on a recording is on-target when one of its ATT&CK tags is in the same technique family as one of the recording's techniques (T1059 / T1059.001 / T1059.003 are one family); a stricter exact-id mode is also available. Rules that fire but are not on-target are counted as off-target alerts: the recordings contain lab background activity and other attack steps, so this is an upper bound on the false-positive burden, not an exact FP rate.

CoverageSummary dataclass

dataset_recall property

datasets instance-attribute

datasets_any_alert instance-attribute

datasets_detected instance-attribute

events instance-attribute

n_rules instance-attribute

off_target_alerts_per_10k instance-attribute

off_target_rules_per_dataset instance-attribute

ruleset instance-attribute

technique_coverage property

Share of techniques with >=1 on-target detection (partial counts as covered).

techniques instance-attribute

__init__(ruleset, n_rules, datasets, datasets_detected, datasets_any_alert, events, off_target_rules_per_dataset, off_target_alerts_per_10k, techniques)

by_tactic()

to_dict()

weighted(weights)

weighted_ci(weights)

95% percentile bootstrap CI of :meth:weighted, resampling techniques (fixed seed).

TechniqueCoverage dataclass

datasets instance-attribute

detected instance-attribute

name instance-attribute

outcome property

rules = field(default_factory=list) class-attribute instance-attribute

tactic instance-attribute

technique instance-attribute

value property

__init__(technique, name, tactic, datasets, detected, rules=list())

channel_ablation(results, rules, channels, kb=None)

Techniques lost if a telemetry channel were not collected (value of each log source).

greedy_rule_selection(results, rules, weights=None, top=10, kb=None)

'Cheapest wins': the few rules that buy the most (weighted) technique coverage.

navigator_layer(summary, attack_version='19', name=None, weights=None)

ATT&CK Navigator layer (format 4.5): green detected / orange partial / red missed.

on_target(rule_techs, ds_techs, exact=False)

score(ruleset, results, rules, kb=None, drop_channels=(), exact=False)

Score replay results. With kb, revoked ATT&CK ids in dataset labels and rule tags (e.g. T1086 -> T1059.001) are mapped to their current replacement first.

gauntlet.sigma

A dependency-light Sigma rule evaluator for Windows event logs.

This is not a SIEM backend: it evaluates SigmaHQ YAML rules directly against JSON event records (e.g. OTRF Security-Datasets / Mordor recordings), which is what a coverage benchmark needs. Supported:

• detection items: field maps (AND of keys, OR of list values), lists of maps (OR), keyword lists (match anywhere in the event), null values
• value modifiers: contains, startswith, endswith, all, windash, re (+ i/m/s), base64, base64offset, utf16le/utf16be/utf16/wide, cased, cidr, exists, gt/gte/lt/lte, fieldref; plain values support */? wildcards
• conditions: and / or / not / parentheses, 1 of X*, all of X*, 1 of them, all of them
• logsources: Sysmon categories, Security 4688 process creation (with field mapping), PowerShell script/module/classic logs and a set of Windows services

Not supported (the rule is reported as unsupported rather than silently passing): aggregations (| count()), correlation rules, expand placeholders, products other than Windows and Linux (Linux: Sysmon for Linux categories and the auditd service only). See docs/adr/0002-sigma-evaluator.md.

SigmaRule dataclass

description = '' class-attribute instance-attribute

id instance-attribute

level instance-attribute

logsource instance-attribute

match = field(repr=False) class-attribute instance-attribute

path = '' class-attribute instance-attribute

references = () class-attribute instance-attribute

status instance-attribute

tactics instance-attribute

targets instance-attribute

techniques instance-attribute

title instance-attribute

__init__(id, title, level, status, techniques, tactics, logsource, targets, match, path='', references=(), description='')

applies(channel, event_id)

cites(pattern)

True if the rule's references/description match pattern (case-insensitive).

load_rules(source, subdir_prefix='')

Load (and memoise per process) Sigma rules; see :func:_load_rules.

gauntlet.replay

Replay recorded telemetry through a rule set (the real-data 'OBSERVE + DETECT' stage).

Instead of executing techniques, GAUNTLET replays public recordings of them (OTRF Security-Datasets) through Sigma rules. Rules are indexed by (channel, EventID) so each event is only tested against rules whose logsource can apply to it.

ReplayResult dataclass

channels instance-attribute

dataset_id instance-attribute

hits = field(default_factory=dict) class-attribute instance-attribute

n_events instance-attribute

pairs = field(default_factory=dict) class-attribute instance-attribute

techniques instance-attribute

__init__(dataset_id, techniques, n_events, channels, hits=dict(), pairs=dict())

fired(drop_channels=())

from_json(d) classmethod

has_target(channel, eid)

Whether the recording contains events a rule targeting (channel, eid) would be tested on.

to_json()

RuleIndex

by_target = defaultdict(list) instance-attribute

errors = Counter() instance-attribute

rules = rules instance-attribute

__init__(rules)

candidates(channel, eid)

evaluate(events, pairs=None)

Evaluate every event; returns (events, per-channel counts, rule -> channel -> hits).

If pairs is given it is updated with per "channel|EventID" event counts.

fingerprint(spec)

Hash of the rule source (file bytes / directory listing) and the evaluator/parser code.

Stored in the replay cache; a mismatch forces a re-replay so bench never reuses stale hits.

load_ruleset(spec)

Rule-set spec: sigma-all:<zip-or-dir>, sigma-core:<zip-or-dir>, sigma-linux:<zip-or-dir> (rules/linux only) or legacy:<dir>.

sigma-core keeps rules with status stable/test and level high/critical -- roughly what a SOC would page on.

replay_dataset(index, ds)

replay_many(spec, datasets, workers=None, cache=None, progress=True)

Replay many datasets in parallel; results cached to cache (JSON) if given.

gauntlet.predict

Technique co-occurrence model: "given what we've seen, what comes next?"

Item-item collaborative filtering over ATT&CK group -> technique sets. For an observed set S, a candidate technique t scores

score(t | S) = sum_{s in S} cooc(s, t) / sqrt(n(s) * n(t))   (cosine)

plus a tiny popularity prior to break ties. Evaluated with leave-one-group-out: hide half of a group's techniques, predict them from the other half using a model trained on every other group, compare with a popularity baseline.

CooccurrenceModel

Item-item cosine co-occurrence model over ATT&CK group technique sets.

co = defaultdict(Counter) instance-attribute

n = Counter() instance-attribute

n_sets = 0 instance-attribute

__init__(sets)

add(s, sign=1)

popularity(k=10, exclude=None)

predict(observed, k=10, exclude=None)

evaluate(sets, ks=(5, 10, 20), hide=0.5, seed=0, min_size=10)

Leave-one-group-out recall@k / MRR, co-occurrence vs popularity.

evaluate_seeds(sets, seeds=range(5), **kw)

Repeat :func:evaluate over several hide-split seeds; report seed 0 plus mean/sd across seeds.

gauntlet.stats

Small, dependency-free uncertainty helpers used by the benchmark.

• :func:wilson -- 95% Wilson score interval for a proportion (coverage, recall).
• :func:pct / :func:fmt_ci -- the single place results are rounded for display.
• :func:bootstrap_ci -- percentile bootstrap CI of the mean of per-unit values (e.g. one value per held-out ATT&CK group).
• :func:paired_bootstrap_ci -- CI of the mean difference between two strategies measured on the same units (like-for-like comparison).
• :func:weighted_bootstrap_ci -- CI of a weighted proportion (threat-weighted coverage), resampling techniques.

All resampling uses a fixed seed so results are reproducible. Nothing here rounds: the generators store full-precision values and the renderers round exactly once (rounding a bound to 4 decimals first and to 1 decimal of a percent later moved some bounds by 0.1).

binom_two_sided(k, n)

Exact two-sided binomial (p=0.5) test p-value: sign test / exact McNemar on n discordant pairs.

bootstrap_ci(values, n_boot=2000, alpha=0.05, seed=0)

Percentile bootstrap CI of the mean of values (fixed seed, n_boot resamples), unrounded.

fmt_ci(ci, digits=1, unit='')

' [lo, hi]' in percent from an unrounded proportion interval, rounded once.

mcnemar_exact(b, c)

Exact McNemar p-value for paired binary outcomes with b and c discordant pairs.

paired_bootstrap_ci(a, b, n_boot=2000, alpha=0.05, seed=0)

Mean of a - b with a bootstrap CI; a and b are aligned per unit.

pct(x, digits=1)

A proportion as a percentage string, rounded once (0.80 -> '80.0').

weighted_bootstrap_ci(hits, weights, n_boot=2000, alpha=0.05, seed=0)

Percentile bootstrap CI of a weighted proportion sum(w*hit)/sum(w), resampling units (techniques).

wilson(successes, n, z=1.959964)

95% Wilson score interval for successes out of n, unrounded.

The bounds are exactly 0.0 when successes == 0 and exactly 1.0 when successes == n.