No live samples by design. Real-file scoring is validated only on benign binaries. Malware performance comes from EMBER's LIEF-extracted features, and VITRINE's own extractor differs from LIEF in small ways (ADR 0002).
Temporal drift. A 2018 model loses AUC and, at its own threshold, most of its recall on EMBER2024 (Evaluation §2). Retrain on recent data for real use. EMBER2024 is used as a ~7 % per-week subsample (43.5 % malicious in test, 50 % in the paper).
The drift ranking is correlational. It ranks features by max(PSI benign, PSI malware) × mean |SHAP|. It does not establish a cause:
2018 → 2024 mixes time with collection (EMBER 2018 was selected to be harder), extractor and schema-translation changes;
parser-based features were extracted by LIEF 0.9 (EMBER 2018) and pefile/thrember (EMBER2024); the both-extractor parity run is described on the Evaluation page, and it covers benign open-source DLLs, not the EMBER files themselves;
the ranking depends on the PSI binning and on family composition (one family, xtrat, is 19 % of the 2018 test malware and absent from the 2024 test);
removing the top-ranked features does not recover robustness, and an oracle alignment of their 2018 marginals lowers 2024 AUC further instead of restoring it, so the shift is not marginal-only;
the alignment test runs on 2018 models retrained in GitHub Actions (AUC within 0.0005 of the committed bench_drift.py models), and the System32 definition parity on a Windows Server 2022 runner, not the Windows 11 host used for the other System32 benchmarks.
Detection is below the authors' published full-data EMBER 2018 model; EMBER 2018 training here is a 50 % subsample, so the gap mixes data and representation.
The LightGBM baselines are memory-limited reproductions (150k rows; 100 trees for the paper config, 300 for the tuned 2018 config). The EMBER 2017 paper setup itself is reproduced exactly (Actions run 37005292317).
The adversarial evaluation works in feature space. It bounds what naive edits can do, and it is not a problem-space attack on real binaries.
YARA structural rules abstain on about half the families, and their lead over a benign-filtered imphash baseline is not significant. String and byte-sequence atoms from code sections (capstone) are not implemented.
The capability floor is not discriminative on its own (sections 5 and 7).