Python API¶
Pipeline¶
End-to-end pipeline: records -> events -> graph -> reduce -> tag -> reconstruct.
Analysis
dataclass
¶
Result of :func:analyze: raw and reduced graphs, alerts, the reconstruction and rejected records.
build_graph(records)
¶
Normalise raw records and build the provenance graph; returns (graph, rejected records).
analyze(records, baseline=None, reduce=True, alert_node=None)
¶
Full pipeline: normalise, build the graph, tag, reduce and reconstruct around the top alert (or alert_node). baseline is a benign capture for the rarity rules.
Loaders¶
Real-data loaders: public provenance / host-telemetry formats -> raw records.
Every loader yields the same raw-record dicts the normalizer already accepts
({"ts", "kind", "pid", ...}), so the graph, tagger and reconstructor are
format-agnostic.
| format | source | module |
|---|---|---|
| sysmon | Sysmon for Linux / Windows XML (OTRF, Splunk attack_data) | sysmon |
| auditd | raw audit.log or AUOMS syslog (OTRF Log4Shell) |
auditd |
| atlas | ATLAS pre-processed, event-labelled audit logs | atlas |
| jsonl | ROOTLINE JSONL / bpftrace probe output (probe, synth) | normalize |
FORMATS = ('auto', 'jsonl', 'sysmon', 'auditd')
module-attribute
¶
sniff(path)
¶
Guess a capture's format from its first 8 KB: sysmon, auditd or jsonl.
merge_sources(*sources, window=1.0)
¶
Fuse record streams from several sensors on the same host(s).
Different sensors see different slices of the same activity (e.g. OTRF's
Log4Shell capture: Sysmon has java's network connections, AUOMS has the
execve whose parent is java). Merging them repairs broken lineage.
Duplicate exec records (same host, pid and image within window
seconds) are kept once - the first source wins - so one exec never
becomes two process vertices. Host names are compared case-insensitively.
load_many(paths, fmt='auto')
¶
Load several captures (formats sniffed unless given) and fuse them with :func:merge_sources.
load_records(path, fmt='auto')
¶
Load one capture as raw records in the given (or sniffed) format.
Models¶
Frozen contracts / typed models for ROOTLINE.
All data flowing between stages uses these types. Keep them stable: the normalizer, graph, detectors, reconstructor and exporters depend on them.
EventKind
¶
Bases: str, Enum
Normalised syscall-level event kinds.
FORK = 'fork'
class-attribute
instance-attribute
¶
EXEC = 'exec'
class-attribute
instance-attribute
¶
OPEN = 'open'
class-attribute
instance-attribute
¶
READ = 'read'
class-attribute
instance-attribute
¶
WRITE = 'write'
class-attribute
instance-attribute
¶
CONNECT = 'connect'
class-attribute
instance-attribute
¶
ACCEPT = 'accept'
class-attribute
instance-attribute
¶
UNLINK = 'unlink'
class-attribute
instance-attribute
¶
EXIT = 'exit'
class-attribute
instance-attribute
¶
DNS = 'dns'
class-attribute
instance-attribute
¶
NodeType
¶
Relation
¶
Bases: str, Enum
Edge types, named in the direction information flows.
FORKED = 'forked'
class-attribute
instance-attribute
¶
EXECUTED = 'executed'
class-attribute
instance-attribute
¶
READ = 'read'
class-attribute
instance-attribute
¶
WROTE = 'wrote'
class-attribute
instance-attribute
¶
CONNECTED = 'connected'
class-attribute
instance-attribute
¶
RECEIVED = 'received'
class-attribute
instance-attribute
¶
DELETED = 'deleted'
class-attribute
instance-attribute
¶
Event
dataclass
¶
A normalized kernel-level event (one syscall of interest).
seq
instance-attribute
¶
ts
instance-attribute
¶
host
instance-attribute
¶
kind
instance-attribute
¶
pid
instance-attribute
¶
ppid = 0
class-attribute
instance-attribute
¶
uid = 0
class-attribute
instance-attribute
¶
comm = ''
class-attribute
instance-attribute
¶
exe = ''
class-attribute
instance-attribute
¶
path = None
class-attribute
instance-attribute
¶
child_pid = None
class-attribute
instance-attribute
¶
dst_ip = None
class-attribute
instance-attribute
¶
dst_port = None
class-attribute
instance-attribute
¶
argv = ()
class-attribute
instance-attribute
¶
sha256 = None
class-attribute
instance-attribute
¶
domain = None
class-attribute
instance-attribute
¶
label = None
class-attribute
instance-attribute
¶
to_dict()
¶
JSON-ready dict.
Node
dataclass
¶
Edge
dataclass
¶
A (possibly merged) edge: count events from ts to end_ts; seq is the first event.
Severity
¶
Alert
dataclass
¶
A tagger hit on a vertex, with ATT&CK technique, kill-chain stage and evidence event ids.
rule_id
instance-attribute
¶
node_id
instance-attribute
¶
ts
instance-attribute
¶
severity
instance-attribute
¶
description
instance-attribute
¶
attack_technique = None
class-attribute
instance-attribute
¶
kill_chain = None
class-attribute
instance-attribute
¶
score = 1.0
class-attribute
instance-attribute
¶
evidence = field(default_factory=list)
class-attribute
instance-attribute
¶
to_dict()
¶
JSON-ready dict.
Reconstruction
dataclass
¶
An attack story: root causes, backward and forward slices, edges, timeline, IOCs, kill chain.
alert
instance-attribute
¶
root_causes
instance-attribute
¶
backward
instance-attribute
¶
forward
instance-attribute
¶
edges
instance-attribute
¶
timeline
instance-attribute
¶
iocs
instance-attribute
¶
kill_chain
instance-attribute
¶
accessed = field(default_factory=set)
class-attribute
instance-attribute
¶
nodes
property
¶
Every vertex in the story (backward, forward and accessed).
Graph¶
Streaming, append-only provenance graph.
Vertices: processes (one vertex per process image: fork and exec each create a new vertex so PID reuse and exec-chains stay distinguishable), files, sockets. Edges point in the direction of information flow, so backward traversal == "what could have influenced this", forward == "what did this influence". Every ingested event extends a SHA-256 hash chain, giving a tamper-evident, append-only record.
GENESIS = '0' * 64
module-attribute
¶
ProvenanceGraph
¶
Append-only provenance graph: one vertex per process image, file or socket, edges in the direction of information flow, every event extending a SHA-256 hash chain.
nodes = {}
instance-attribute
¶
edges = []
instance-attribute
¶
out_edges = defaultdict(list)
instance-attribute
¶
in_edges = defaultdict(list)
instance-attribute
¶
events = []
instance-attribute
¶
head = GENESIS
instance-attribute
¶
dns = defaultdict(set)
instance-attribute
¶
proc_of(ev)
¶
Current process-image vertex of pid on host (created on first sight).
file_node(host, path, ts)
¶
Vertex for a file path (one per host and path).
sock_node(ip, port, ts)
¶
Vertex for a remote endpoint ip:port.
add_event(ev)
¶
Add one normalised event: update vertices and edges and extend the hash chain.
ingest(events)
¶
Add events in order and return the graph (chainable).
find(needle, ntype=None)
¶
Vertex ids whose label or id contains needle (optionally of one type).
stats()
¶
Counts of events, vertices, edges and vertices per type.
chain_hash(prev, ev)
¶
Next hash-chain head: SHA-256 over the previous head and the event's canonical JSON.
verify_chain(events, expected_head)
¶
True when replaying events reproduces head (any edit, drop or reorder breaks it).
Reduction¶
Graph reduction: collapse benign repetition while preserving causality.
Two heuristics (simplified from the provenance-reduction literature, e.g. Xu et al. "CPR" CCS'16 and LogGC):
- Edge merge - repeated identical (src, dst, rel) edges are merged into
one edge with
count/last_ts, unless the source vertex received new information in between (an in-edge whose timestamp falls inside the gap), which would change what flows (the CPR causality-preservation rule). - Benign leaf pruning - read-only system files (shared libs, locale,
/proc, ld cache) that nothing wrote during capture are dropped; they are
noise for attack stories. Anything in
keepis never pruned.
BENIGN_READ_PATTERNS = [re.compile(p) for p in ('^/(usr/)?lib(32|64)?/', '\\.so(\\.\\d+)*$', '^/proc/', '^/sys/', '^/etc/ld\\.so\\.cache$', '^/usr/share/(locale|zoneinfo|fonts|icons)/', '^/etc/(localtime|nsswitch\\.conf|hosts|resolv\\.conf)$', '^/dev/(null|urandom|tty)')]
module-attribute
¶
ReductionReport
dataclass
¶
is_benign_readonly(g, nid)
¶
True for read-only library, locale and config files that carry no attack information.
reduce_graph(g, keep=None)
¶
Causality-preserving reduction (CPR-style edge merging, benign read-only leaf pruning). Vertices in keep are never pruned. Returns (reduced graph, report).
Reconstruction¶
Backward root-cause + forward blast-radius reconstruction.
Traversal is time-respecting: walking backward from a vertex observed at time t only follows in-edges that happened at or before t (information cannot flow from the future), and forward traversal only follows out-edges at or after the time influence arrived. This is the standard trick (King & Chen, "Backtracking Intrusions", SOSP'03) that keeps the dependency explosion in check. Pure graph traversal - no ML - so every step is explainable.
STOP_COMMS = {'systemd', 'init', 'sshd', 'gnome-session', 'gdm', 'lightdm', 'cron', 'kthreadd', 'login', 'services.exe', 'wininit.exe', 'winlogon.exe', 'smss.exe', 'csrss.exe', 'explorer.exe', 'svchost.exe', 'lsass.exe', 'system'}
module-attribute
¶
ENTRY_EXT = re.compile('\\.(docm?|xlsm?|pptm?|pdf|rtf|odt|zip|rar|7z|iso|lnk|js|hta|jar|sh|py|elf|bin|deb)$', re.I)
module-attribute
¶
USER_DIRS = re.compile('^/(home/[^/]+|root)/(Downloads|Desktop|Documents|tmp)/|^/tmp/|^c:/users/[^/]+/(downloads|desktop|documents|appdata/local/temp)/', re.I)
module-attribute
¶
SYSTEM_PATHS = re.compile('^/(usr|bin|sbin|lib|etc|opt|proc|sys)/|^/dev/(null|zero|u?random|full|tty\\w*|ptmx|pts/)|^c:/(windows|program files|programdata)/', re.I)
module-attribute
¶
APP_STATE = re.compile('/(\\.mozilla|\\.config|\\.cache)/|/appdata/(roaming|locallow)/|/appdata/local/(?!temp/)', re.I)
module-attribute
¶
NON_IOC_PATHS = re.compile('^/(dev|proc|sys|run)/')
module-attribute
¶
STAGE_ORDER = ['initial-access', 'delivery', 'execution', 'persistence', 'privilege-escalation', 'defense-evasion', 'credential-access', 'discovery', 'lateral-movement', 'collection', 'command-and-control', 'exfiltration', 'impact']
module-attribute
¶
is_local_ip(ip)
¶
Loopback, unspecified or link-local (IPv4, IPv6 and IPv4-mapped IPv6): never an external IOC.
backward(g, start, t, max_nodes=5000, stop=None, timed=True)
¶
Time-respecting ancestors of start as of time t.
Session/service roots in stop (default :data:STOP_COMMS) are not expanded.
timed=False ignores time (plain reachability); it exists for the ablation study.
forward(g, start, t, max_nodes=5000, stop=None, timed=True)
¶
Time-respecting descendants of start. Session/service roots in stop
(default :data:STOP_COMMS) are included but not expanded: a system service
that merely touched a malicious file (indexer, AV, crash reporter) must not
pull its entire lifetime into the blast radius.
score_entry(g, nid)
¶
Heuristic entry-point score of a file or socket vertex (higher = more likely an attack entry).
root_causes(g, back, back_edges, k=3, t=float('inf'))
¶
Rank candidate entry points: files/sockets in the backward slice.
A candidate must have fed information into the slice strictly before the
pivot time t: a C2 socket the pivot process itself opened is an effect
of the intrusion, not its cause.
contact_window(g, nid)
¶
First and last time anything flowed into or out of a vertex.
reconstruct(g, alert, alerts=None, seeds=None, *, timed=True, stops=True, spine=True, accessed=True)
¶
Reconstruct around alert (backward from, and forward after, alert.ts).
seeds = extra (vertex, t_backward, t_forward) pivots traced together
with the alert - used for IOC pivots, where an attacker IP maps to several
socket vertices and influence starts at first contact, not last.
The keyword switches turn off one component each, for the ablation study
(rootline.ablation): timed (time-respecting traversal), stops
(session-root stops), spine (causal-spine trimming of the backward slice)
and accessed (data read by intrusion-created processes and executed images).
Export¶
Exporters: JSON attack story (REVENANT handoff), STIX 2.1 bundle, Mermaid.
STIX objects are built by hand (spec-shaped dicts) to avoid a heavy
dependency; the test suite validates them with python-stix2.
All labels come from untrusted telemetry. Every exporter escapes them for its target language: Cypher string literals (no data in comments), Mermaid entity codes, JSON via the json module.
NS = uuid.UUID('6f1b3c5e-0000-4000-8000-526f6f746c6e')
module-attribute
¶
to_story(g, r)
¶
Return the rootline.story/v1 JSON: nodes, edges, timeline, IOCs and the integrity head.
to_stix(g, r)
¶
Return a STIX 2.1 bundle (identity, IP/file observables, indicators, attack-pattern).
to_mermaid(g, r)
¶
Return the story as a Mermaid flowchart (root causes outlined red, alert vertex filled).
dumps(obj)
¶
Pretty JSON with a str fallback for non-JSON types.
to_cypher(g, r=None)
¶
Neo4j import script (cypher-shell < story.cypher).
Exports the reconstructed story (or the whole graph when r is None) as
idempotent MERGE statements: (:Process|:File|:Socket {id}) vertices
and typed relationships (:FORKED, :WROTE ...) carrying ts/count.