A controlled, leakage-free benchmark of six one-class classifiers for network intrusion detection — trained on benign traffic only, thresholded on a held-out validation split, and evaluated once on frozen test sets from UNSW-NB15 and CIC-IDS-2018.
Each line is one detector. Left: its AUC-ROC rank on UNSW-NB15. Right: its rank on CIC-IDS-2018. If benchmark results generalized, the lines would be horizontal. They are not.
A large share of NIDS papers report results on a single dataset and imply the conclusions travel. This benchmark measures how far they actually travel: under identical preprocessing, splits and thresholding, only one detector keeps its position.
// hover or tap a line for details
AUC-ROC gives the threshold-independent view; F1 is measured at the Youden's J threshold frozen on validation. The two views need not agree — on UNSW-NB15 the Autoencoder wins AUC while Isolation Forest takes the best F1.
| UNSW-NB15 | AUC | F1 | Rank |
|---|
| CIC-IDS-2018 | AUC | F1 | Rank |
|---|
MCD on CIC-IDS-2018 is the cautionary row: F1 0.8413 next to AUC 0.6288 — a lucky operating point on a weak score. Reporting F1 alone would have hidden it.
Detection rate per attack category on UNSW-NB15, at each detector's frozen operating point. The global winner misses what weaker detectors catch — 82% of Reconnaissance and 94% of Shellcode slip past the Autoencoder.
MCD's higher category recall comes at the price of a much higher global false-positive rate — there is no free lunch at a single operating point.
Every detector runs through the same pipeline. Nothing downstream of the split ever sees test data, and nothing in preprocessing ever sees an attack.
Benign traffic split into train / validation before any statistic is computed. Test sets frozen and untouched until the final run.
log1p transform, frequency encoding and quantile clipping fitted exclusively on benign training data — no attack ever leaks into preprocessing.
Six detectors — Autoencoder, Deep SVDD, Isolation Forest, OC-SVM, PCA-RE, MCD — learn only what "normal" looks like.
Youden's J selected on a held-out validation split, then frozen. Test-time metrics are measured at that operating point, never tuned on test.
Why the ceremony? Several published NIDS results tune thresholds on the test set, which inflates operating-point metrics. Freezing the threshold on validation makes the reported F1, FPR and FNR honest estimates of deployment behaviour — the numbers here are lower than some published figures by design.
The Autoencoder is the only detector in the top two on both datasets. Its inductive bias — learn whatever non-linear structure normal traffic has — is the least committal, and it generalizes best.
Deep SVDD partially collapses on CIC-IDS-2018: correlated features let the encoder map everything near the centre. The failure is invisible without inspecting training curves — the AUC still looks respectable.
Averaging normalized scores across all six detectors loses to the best single model on both datasets. Weak members drag the score down; diversity alone is not a strategy.
Even the best detector's 6% FPR means roughly 60,000 false alerts per million benign flows per day. Anomaly-based NIDS is a triage layer, not a verdict.
OC-SVM needs ~400 s to train on CIC-IDS-2018 where PCA-RE needs 2.3 s — for a worse AUC. Compute cost and detection quality are uncorrelated in this benchmark.
AUC and F1 crown different winners on UNSW-NB15, and F1 alone flatters MCD on CIC-IDS-2018. Threshold-independent and operating-point metrics must be reported together.