Network Intrusion Detection Eval

network-intrusion-detection-eval

Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives — Divekar et al. (2018) (arXiv:1811.05372, 2018)

What this evaluates

Evaluates the ability of machine learning classifiers to detect network intrusions across different traffic datasets, with a focus on handling severe class imbalance and identifying rare attack types.

Datasets

• KDD-99 — total ?; splits: test (-1)
• NSL-KDD — total ?; splits: test (-1)
• UNSW-NB15 — total ?; splits: test (-1)

Metrics

• Weighted F1-Score (primary) — range: percent
  • The average of per-class F1-scores weighted by the number of true instances (support) for each class. Calculated as sum(w_i * F1_i) / sum(w_i), where w_i is the support of class i.

Input / output format

Input: Network traffic records represented as feature vectors containing numerical and categorical attributes describing connection properties (e.g., duration, protocol, service, flag, byte counts).

Output: Predicted class label from a predefined set of intrusion categories (e.g., DoS, Normal, Probe, R2L, U2R) or a binary attack/normal label.

Scoring recipe

def compute_weighted_f1(y_true, y_pred, classes):
    f1s = []
    weights = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
        weights.append(sum(1 for t in y_true if t == c))
    return sum(w * f for w, f in zip(weights, f1s)) / sum(weights)

Common pitfalls

• Using raw accuracy as the primary metric, which is misleading due to extreme class imbalance in network traffic data.
• Comparing multi-class F1-scores directly across datasets with different class taxonomies without binarization or proper weighting.
• Overlooking minority attack classes (e.g., R2L, U2R, Shell Code) which are critical for security but often have near-zero scores on legacy datasets.

Evidence (verbatim from paper)

Unlike other works [[17]], we consider primarily the Weighted F1-Score, as accuracy is misleading for such data distributions.

Citation

@misc{divekar2018benchmarking,
  title={Benchmarking datasets for Anomaly-based Network Intrusion Detection: KDD CUP 99 alternatives},
  author={Divekar et al. (2018)},
  year={2018},
  note={arXiv:1811.05372}
}

• arXiv: 1811.05372