Evaluation

Methodology, every result table and its confidence interval. Generated from the README by scripts/sync_docs.py.

Headline results

Numbers are the mean over 5 seeds from results/<dataset>/seeds.json (GitHub Actions ubuntu runners; full 95 % Student-t CIs, n=5, t(4)=2.78, clipped to [0, 1], in CIC seeds.md and UNSW seeds.md; the CIC 10 % subsample is redrawn per seed). Attack budgets eps are L-inf radii on the attacker-controlled features in standardised log-feature space (ADR 0002); derived features are recomputed after projection and can move further (see Limitations). Detection rates are on 1,000 held-out attack flows per seed; every constrained adversarial flow passed the schema validity check. Hardened tree models and ensembles are attacked through both the undefended and the adversarially-trained MLP as gradient surrogates plus 100 black-box queries, keeping the per-flow best.

Detection rate under constrained, adaptive evasion (higher is better), 5-seed mean; eps=2 with 95 % CI

model	CIC-IDS2017 eps=0	eps=0.5	eps=1	eps=2	UNSW-NB15 eps=0	eps=0.5	eps=1	eps=2
MLP	0.991	0.531	0.308	0.058 [0.001, 0.116]	0.964	0.436	0.181	0.064 [0.052, 0.077]
Random forest (baseline)	0.995	0.456	0.421	0.377 [0.306, 0.447]	0.971	0.656	0.152	0.000 [0.000, 0.000]
XGBoost	0.998	0.450	0.434	0.405 [0.376, 0.433]	0.967	0.167	0.145	0.000 [0.000, 0.000]
FEINT ensemble (XGB + MLP, OR autoencoder)	0.998	0.557	0.469	0.369 [0.279, 0.459]	0.969	0.502	0.445	0.328 [0.262, 0.394]
MLP + adversarial training	0.986	0.814	0.753	0.688 [0.580, 0.797]	0.961	0.927	0.888	0.808 [0.746, 0.869]
XGBoost + adversarial training	0.999	0.807	0.805	0.788 [0.755, 0.821]	0.967	0.960	0.959	0.941 [0.908, 0.974]
FEINT ensemble + adversarial training	0.998	0.839	0.825	0.793 [0.763, 0.823]	0.969	0.965	0.964	0.962 [0.957, 0.968]
XGBoost, robust features only	0.994	0.994	0.994	0.994 [0.993, 0.995]	0.942	0.942	0.942	0.942 [0.935, 0.950]

What the numbers say:

1. The collapse is real, and textbook attacks overstate it. Textbook (unconstrained, all-feature) attacks drive the undefended MLP, XGBoost and ensemble (the three models attacked that way) to 0.000-0.044 detection on CIC-IDS2017 at eps=0.5 (seed 0), but produce 0 % realisable flows. Under the realistic constraints the undefended XGBoost keeps 42 %, almost entirely volume attacks (DoS Hulk 94 %, GoldenEye 64 %, DDoS 57 %) whose server-side features the attacker cannot touch; PortScan, slowloris, Slowhttptest, FTP/SSH-Patator, Bot and web attacks all go to 0 %. The two attacks are not budget-matched, so neither is an upper bound of the other: the constrained attack moves only controllable features by at most eps but the recomputed derived features can move further, while the textbook attack moves every feature by at most eps (on UNSW-NB15 the ensemble keeps 0.353 at eps=2 against the textbook attack; see Limitations).
2. The most important feature is attacker-controlled. TreeSHAP ranks init_win_fwd (client TCP window) first on CIC-IDS2017 and sttl (source IP TTL, a known UNSW-NB15 testbed artefact) first on UNSW-NB15. Controlling only sttl drops UNSW XGBoost detection to 0.2 %; controlling only init_win_fwd drops CIC detection to 48 %.
3. Hardening works at little clean cost. Adversarial training keeps clean F1 within 0.002 (5-seed means) and restores constrained detection at eps=2 to 0.79 (CIC) and 0.94-0.96 (UNSW) for XGBoost and the ensemble, 0.69 / 0.81 for the MLP. Restricting the model to the features the schema fixes makes it immune to this attacker by construction, for a clean FPR of 2.6 % instead of 0.4 % on CIC and 33 % instead of 26 % on UNSW; that immunity holds only if those features really are out of the attacker's reach (they are not entirely, see Limitations). On CIC-IDS2017 the ensemble and XGBoost cannot be told apart at eps=2 (0.369 vs 0.405, CIs overlap), and the ensemble pays about 4x the FPR.
4. Base rates matter more than the leaderboard. XGBoost's 99.7 % accuracy and 0.40 % [0.36, 0.44] FPR (5 seeds) mean only 20 % [19, 22] of its alerts are real at a 0.1 % attack prevalence; on UNSW-NB15 (26 % FPR on the official test split) that falls to 0.4 %.

study	CIC-IDS2017	UNSW-NB15
Counterfactuals (seed 0, single attack at eps_max=2, 50 queries): XGBoost alerts with a realisable evasion	60.0 % of 200 (2.07 features changed, median eps 0.031)	99.5 % of 200 (2.09 features)
...same, against the adversarially-trained ensemble (a different model; lower bound, weaker attack than the curves)	7.5 %	0.5 %
Adversarial-input alarm, static attacker (ROC-AUC / FPR on clean)	1.000 / 1.2 %	1.000 / 1.7 %
...detector-aware attacker vs. XGBoost OR alarm, single attack at eps=2 (not a union over budgets)	detection 0.876, benign FPR 2.8 %	detection 1.000, benign FPR 28.3 %
Backdoor, 2 % of a 100k-flow training subsample poisoned: success clean -> poisoned -> sanitised (5 seeds)	0.33 [0.32, 0.34] -> 1.00 -> 0.08 [0.02, 0.14]	0.04 [0.02, 0.05] -> 1.00 -> 0.68 [0.62, 0.74]
Sanitiser poison recall / precision (5 seeds)	0.99 / 0.51	0.95 / 0.36
Sanitiser cost: clean benign flows removed / clean accuracy	2.6 % / 0.996 -> 0.985	10.8 % / 0.866 -> 0.840

Drift (CIC-IDS2017, train Monday-Wednesday, test Thursday-Friday). Supervised models do not generalise to unseen attack families: XGBoost recall is 0.5 % on PortScan (n=9,074), 0 % on Bot, 0 % on Infiltration. The benign-trained autoencoder catches 29 of 36 Infiltration flows (81 %, Wilson 95 % CI 65-90 %), which is why the ensemble ORs it in (at 3.2 % FPR instead of 0.04 %).

Full results

Complete generated reports, including per-family tables, SHAP rankings, single-feature exploitability, counterfactual statistics, poisoning and drift tables: CIC-IDS2017 · UNSW-NB15.

Clean detection vs. baselines and published numbers (seed 0, report.json)

model	CIC acc	CIC F1	CIC FPR	CIC prec @0.1 %	UNSW acc	UNSW F1	UNSW FPR
Logistic regression	0.8925	0.7833	0.0607	0.012	0.8128	0.8512	0.3832
Random forest	0.9963	0.9928	0.0032	0.236	0.8643	0.8874	0.2661
IsolationForest (unsupervised)	0.7420	0.0420	0.0093	0.002	0.4459	0.0108	0.0146
Autoencoder (unsupervised)	0.7611	0.1751	0.0100	0.010	0.5857	0.4167	0.0259
MLP	0.9925	0.9854	0.0064	0.134	0.8566	0.8817	0.2822
XGBoost	0.9970	0.9942	0.0035	0.220	0.8654	0.8879	0.2605
FEINT ensemble	0.9892	0.9794	0.0134	0.070	0.8566	0.8820	0.2856
FEINT ensemble + adv. training	0.9889	0.9789	0.0137	0.068	0.8605	0.8844	0.2720
XGBoost, robust features only	0.9794	0.9612	0.0255	0.037	0.8159	0.8485	0.3324
Published: RF, Sharafaldin et al. 2018, Table 4 (per-attack selected features of Table 3, weighted multi-class P/R/F1; split not reported)		0.97
Published: decision tree, Moustafa & Slay 2016 1 (FAR 15.78 %)					0.8556

On CIC-IDS2017 with 11 of the ~80 CICFlowMeter features and de-duplicated data, XGBoost reaches binary F1 0.994; on the UNSW-NB15 official split XGBoost reaches 86.6 % accuracy vs. the published 85.6 % decision tree (see the reproduction table for a like-for-like run). Published numbers use different feature sets and preprocessing and are shown for orientation, not as a controlled comparison. The Sharafaldin et al. figure (RF: Pr 0.98, Rc 0.97, F1 0.97) was checked against Table 4 of the ICISSP 2018 paper; the Moustafa & Slay figure (DT: 85.56 % accuracy, 15.78 % FAR) is unverified: the paper is paywalled, and no legitimate open copy (author page, UNSWorks, the UNSW-NB15 dataset page) or open secondary source restating these numbers could be found (searched 2026-10; arXiv 1811.05372, 2008.07405, 2009.13067 and 1711.02825 cite the paper but do not restate them). Treat it as an unconfirmed orientation figure.

CIC-IDS2017 global importance	CIC-IDS2017 drift

New datasets, corrected labels and cross-dataset transfer

All runs below are GitHub Actions jobs (.github/workflows/bench.yml), never the shared laptop.

Corrected CIC-IDS2017 (Engelen et al. 2021; the CNS 2022 re-release by Liu et al., which also fixes CSE-CIC-IDS2018). With labelling and flow-construction errors fixed and "attempted" flows relabelled benign, clean scores rise (XGBoost F1 0.9999, FPR 0.05 %) and so does robustness: over 5 seeds undefended XGBoost keeps 0.523 [0.490, 0.556] detection at eps=2 (0.405 on the original), random forest falls to 0.005, adversarially trained models stay at 0.980-0.997, and the poison sanitiser's precision rises from 0.51 to 0.94 (seeds.md, report). Part of the original dataset's apparent difficulty comes from mislabelled flows.

CSE-CIC-IDS2018 and cross-dataset transfer. Three day files (1.09 GB; Bot, FTP/SSH brute force, DoS GoldenEye/Slowloris), SHA-256 pinned from a size-verified CI download. Training on corrected CIC-IDS2017 and testing on 2018 over the harmonised 11-feature schema and shared families, 5 seeds (results): XGBoost F1 0.000 [0.000, 0.000], adversarially trained XGBoost 0.211, robust-feature XGBoost 0.506 (SSH brute force recall 0.994, the rest near 0). The reverse direction gives 0.015 / 0.488 [0.078, 0.898] / 0.024. Hardening changes what transfers, but nothing transfers well: the detectors learn dataset artefacts, not attack behaviour.

CICIoT2023 (subsample, negative result). A public ipfixprobe Parquet re-export, one capture per class, at most 5,000 flows per label (16 GB RAM); not comparable with the official CSV features of Neto et al. 2023. With these features benign and attack captures are not separable: every model reaches F1 0.98 only by flagging almost everything (benign FPR 0.82-0.997, 5 seeds, seeds.md). We report it as a failed transfer of the pipeline to IoT data, not as a detector.

Schema ablation: what the schema buys

Two paired ablations (5 seeds; each seed shares split, preprocessor, poison set and evaluation attack between arms; difference = schema arm minus schema-agnostic arm, Student-t 95 % CI over the per-seed differences; CIC, UNSW).

comparison	metric	CIC-IDS2017	UNSW-NB15
MLP adversarially trained on constrained vs unconstrained (textbook) examples, same budget	constrained detection at eps=2	0.676 vs 0.031, diff +0.645 [+0.529, +0.762]	0.808 vs 0.080, diff +0.728 [+0.665, +0.792]
...same	clean FPR	0.008 vs 0.007, diff +0.001 [-0.002, +0.003]	0.276 vs 0.280, diff -0.004 [-0.017, +0.010]
kNN poison sanitiser in robust-feature space vs all features (Paudice et al. 2018 style)	backdoor success after sanitising	0.083 vs 1.000, diff -0.917 [-0.976, -0.858]	0.676 vs 0.903, diff -0.227 [-0.311, -0.144]
...same	clean accuracy	0.985 vs 0.996, diff -0.011 [-0.012, -0.010]	0.840 vs 0.845, diff -0.006 [-0.007, -0.004]

At eps=2, training on textbook examples buys almost nothing against the realisable attacker, while schema-projected examples recover most of the detection at no FPR cost. The robust-feature restriction is what makes the sanitiser find the poison on CIC-IDS2017 (poison recall 0.99 vs 0.09); on UNSW-NB15 it helps but leaves a 68 % backdoor. The sanitiser pays about one accuracy point for it. Both gains are conditional on the schema being right (see Limitations).

Reproductions: paper vs reproduction vs FEINT

paper (setup)	metric	paper	our reproduction under the paper's setup	FEINT (own setup)
Moustafa & Slay 2016, decision tree, official UNSW-NB15 split, all 42 features (json)	accuracy / FAR	0.8556 / 0.1578 (unverified, see note)	0.8649 / 0.2484	XGBoost, 22 features: 0.8655 / FPR 0.259
Sharafaldin et al. 2018 Table 4, RF on the union of the Table 3 per-attack selected features, weighted multi-class P/R/F1, split not reported	F1	0.97	RF 0.999 (KNN 0.995, ID3 0.998, AdaBoost 0.888, MLP 0.981, NB 0.141; QDA with defaults cannot be fitted, QDA with reg_param=1e-3 and the 3 classes with fewer rows than features dropped from training: 0.869) (json); on corrected CIC-IDS2017 (Engelen et al. 2021): RF 0.995, KNN 0.993, ID3 0.995, AdaBoost 0.954, MLP 0.994, NB 0.365 (json)	binary F1 0.994 on 11 de-duplicated schema features; not like-for-like
Vitorino et al. 2022 (A2PM, adversarial NIDS), CIC-IDS2017 Tuesday + Wednesday, 8 classes, stratified 70/30, the paper's RF and Keras MLP, regular and A2PM adversarial training, 50-iteration A2PM attacks, seed 0 (json, report)	accuracy on malicious flows after 50 iterations, targeted / untargeted; untargeted macro-F1	RF 0.000 / 0.000 / 0.209; RF + adv. training 0.999 / 0.900 / 0.543; MLP 0.104 / 0.009 / 0.183; MLP + adv. training 0.944 / 0.789 / 0.510 (Figures 5-7)	RF 0.000 / 0.000 / 0.234; RF + adv. training 0.932 / 0.901 / 0.642; MLP 0.404 / 0.078 / 0.166; MLP + adv. training 0.947 / 0.931 / 0.603	binary detection of all CIC-IDS2017 attack days under FEINT's realisable attack at eps=2 (5 seeds): XGBoost 0.405, XGBoost + adv. training 0.788; not like-for-like

Accuracy reproduces within one point; our false-alarm rate is far above the quoted 15.78 %, which we could not verify (the paper is paywalled and no open copy or open secondary source was found). The gap is not an obvious definition mismatch: for our tree, the mean of FPR and FNR is 0.146, the false-discovery rate 0.175 and the error rate 0.135, none of them 0.158. We therefore treat both published figures as unverified.

Our Sharafaldin reproduction (stratified 70/30 split, duplicates kept, scikit-learn defaults; split and hyperparameters are our assumptions) has a higher F1 than Table 4 for every classifier that fits: RF 0.999 vs 0.97, and AdaBoost/MLP about 0.89/0.98 vs 0.77/0.76. A random split with duplicates kept leaks near-identical flows into the test set, which is the most likely reason; QDA with defaults fails because the benign covariance matrix is singular on these features; a regularised QDA (a deviation, labelled as such) reaches 0.869 against the paper's 0.92, the only classifier below its published F1. On the corrected release the same setup gives the same picture (every fitted model above Table 4), so label noise does not explain the gap.

A2PM (Vitorino et al. 2022) reproduces in shape, and partly in value. Regularly trained models collapse under 50 A2PM iterations and A2PM adversarial training keeps most of the accuracy, as in the paper. Six of the twelve end points agree within 0.03. For example, the RF reaches 0.000 accuracy on malicious flows under both attacks (paper 0.000), the adversarially trained RF keeps 0.901 under the untargeted attack (paper 0.900) and the adversarially trained MLP keeps 0.947 under the targeted one (paper 0.944). The largest differences are in the MLPs. Our regular MLP resists the targeted attack longer: 0.404 after 50 iterations against 0.104, although its first-iteration drop of 16 points matches the paper's 15. Our adversarially trained MLP keeps 0.931 under the untargeted attack against 0.789. Our adversarially trained models also keep a higher untargeted macro-F1 (0.64 / 0.60 against 0.54 / 0.51). Our adversarially trained RF loses 7 points under the targeted attack, where the paper's loses none. Our MLP's clean macro-F1 is 0.88 against the paper's 0.97, because its minority classes are weaker. We know of two deviations. The MachineLearningCVE CSVs carry the destination port as one numeric column; the paper one-hot encoded port and protocol. We also replaced a2pm 1.2.0's per-value Python loops (hours per attack) with vectorised subclasses that draw from the same distribution. Every run checks them against the library, at transform level and end to end: the largest accuracy difference was 0.0017 over 4 seeds. An earlier run with the library's own transforms gives the same RF results (cross-check). Single seed, single split.

Model stealing (label-only queries -> transfer attack), seed 0

surrogate	CIC-IDS2017 agreement	victim det. eps=0.5 / 1 / 2	UNSW-NB15 agreement	victim det. eps=0.5 / 1 / 2
true labels, full training set	0.995	0.631 / 0.612 / 0.567	0.937	0.544 / 0.400 / 0.245
stolen, 500 label queries	0.837	0.529 / 0.520 / 0.445	0.785	0.568 / 0.329 / 0.178
stolen, 2000 label queries	0.956	0.525 / 0.504 / 0.452	0.914	0.435 / 0.360 / 0.001
stolen, 10000 label queries	0.989	0.547 / 0.518 / 0.478	0.918	0.514 / 0.422 / 0.001

With 2,000 hard-label queries a stolen MLP transfers at least as well as one trained on the true labels at eps=2 on both datasets; at smaller budgets it depends on the budget. That the stolen copy imitates the victim's boundary is an untested hypothesis. Single seed.

---

1. N. Moustafa, J. Slay. The evaluation of Network Anomaly Detection Systems: Statistical analysis of the UNSW-NB15 data set and the comparison with the KDD99 data set. Information Security Journal: A Global Perspective 25(1-3), 2016. ↩