Benchmarks & results¶

All numbers below are produced by the scripts in benchmarks/ from the public corpora described in Datasets, and rendered by python benchmarks/make_figures.py. The raw JSON lives in results/*.json; the methodology is ADR 0006.

Reproduce

pip install -e '.[bench,evtx]'
python scripts/download_data.py
python benchmarks/bench_edges.py && python benchmarks/bench_stories.py
python benchmarks/bench_antiforensics.py && python benchmarks/bench_scale.py
python benchmarks/make_figures.py     # writes results/*.png and RESULTS.md

Figures¶

Causal-edge F1 by effect type — B1 — causal-edge F1 vs the PID-nearest baseline, per effect type.

Confidence calibration — Edge-confidence calibration: hand-set weights vs cross-corpus calibration.

Story hit@k — B2 — story ranking hit@k vs flat timelines.

Rule-engine scaling — B4 — rule-engine scaling: v0.1 (quadratic) vs v0.2 (near-linear).

Full results table¶

REVENANT benchmark results¶

Generated by python benchmarks/make_figures.py from results/*.json. Do not edit by hand.

B1 - causal edges vs Sysmon GUID ground truth¶

corpus	method	precision	recall	F1	false-edge rate
otrf_atomic	v0.1 exact-ref join	0.518	0.236	0.324	0.482
otrf_atomic	PID-nearest baseline	0.957	0.846	0.898	0.043
otrf_atomic	REVENANT v0.2	0.955	0.955	0.955	0.045
otrf_apt29_day1	v0.1 exact-ref join	0.500	0.126	0.202	0.500
otrf_apt29_day1	PID-nearest baseline	1.000	1.000	1.000	0.000
otrf_apt29_day1	REVENANT v0.2	1.000	1.000	1.000	0.000

Per effect type (REVENANT vs PID-nearest, F1):

corpus	effect	n	PID-nearest	REVENANT
otrf_atomic	dns_query	105	0.804	0.952
otrf_atomic	file_delete	450	0.826	0.878
otrf_atomic	file_read	280	0.933	1.000
otrf_atomic	file_time_change	137	0.985	0.993
otrf_atomic	file_write	1671	0.947	0.986
otrf_atomic	network_connect	206	0.731	0.961
otrf_atomic	pipe	350	0.856	0.937
otrf_atomic	process_access	10609	0.915	0.927
otrf_atomic	process_start	321	0.804	0.878
otrf_atomic	registry_set	52090	0.894	0.961
otrf_atomic	remote_thread	6	0.909	1.000
otrf_apt29_day1	dns_query	72	1.000	1.000
otrf_apt29_day1	file_delete	367	1.000	1.000
otrf_apt29_day1	file_read	179	1.000	1.000
otrf_apt29_day1	file_time_change	205	1.000	1.000
otrf_apt29_day1	file_write	1283	1.000	1.000
otrf_apt29_day1	network_connect	467	1.000	1.000
otrf_apt29_day1	pipe	225	1.000	1.000
otrf_apt29_day1	process_access	15553	1.000	1.000
otrf_apt29_day1	process_start	327	1.000	1.000
otrf_apt29_day1	registry_set	60927	1.000	1.000
otrf_apt29_day1	remote_thread	2	1.000	1.000

Edge-confidence calibration (ECE, lower is better)¶

evaluated on	hand-set	cross-corpus calibrated
otrf_atomic	0.245	0.035 (fit on the other corpus)
otrf_apt29_day1	0.274	0.013 (fit on the other corpus)

B2 - story ranking vs flat timelines (OTRF atomic, ATT&CK labels from metadata)¶

method	hit@1	hit@3	hit@5	captures reached	median events read	mean events read
chronological timeline	0.000	0.000	0.000	54/91	204.0	1812.8
flat, suspicion-sorted	0.396	0.472	0.560	54/91	0.0	3.6
REVENANT stories	0.440	0.527	0.549	51/91	1	8

B3 - anti-forensics on EVTX-ATTACK-SAMPLES (file level)¶

278 readable .evtx files, 10 labelled as log/timestamp tampering by file name. Parser: 37,364 records; 58.1% of the 4,846 records in supported channels mapped (7.5% of all records, most of the rest being one ETW RPC trace); 21 records/s (python-evtx).

detector	precision	recall	F1	TP	FP	FN
1102/104 query (baseline)	0.115	0.300	0.167	3	23	7
REVENANT, all indicators	0.172	0.500	0.256	5	24	5
REVENANT, high severity only	0.172	0.500	0.256	5	24	5

Missed positives: DE_EventLog_Service_Crashed.evtx, de_PsScriptBlockLogging_disabled_sysmon12_13.evtx, DE_remote_eventlog_svc_crash_byt3bl33d3r_sysmon_17_1_3.evtx, DE_WinEventLogSvc_Crash_System_7036.evtx, Sysmon_12_DE_AntiForensics_MRU_DeleteKey.evtx

B4 - scale (APT29 day 1, full capture)¶

128,095 normalized events from 196,081 raw rows (mapping coverage 65.3%)
analysis 108.35 s (1,182 events/s); stages: {'ingest': 14.9797, 'fusion': 7.285, 'rules': 28.0202, 'antiforensics': 31.15, 'stories': 26.3879}
93,974 causal edges, 1,475 cross-artefact corroborations, 50 stories, 209 tamper/coverage indicators
rule engine log-log slope: v0.1 2.176 vs v0.2 1.033; v0.1 extrapolated to the full capture: ~241,256 s