Cicids2017 Adversarial Eval

cicids2017-adversarial-eval

An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection — Vitorino et al. (2024) (arXiv:2402.16912, 2024)

What this evaluates

Evaluates the adversarial robustness of tree ensemble models (RF, XGB, LGBM, EBM) on enterprise network intrusion detection using the CICIDS2017 dataset. It measures how well models maintain detection performance on benign and malicious traffic when subjected to constrained adversarial perturbations of time-series traffic features.

Datasets

• CICIDS2017 — total ?; splits: test (-1)

Metrics

• F1S (primary) — range: percent
  • Harmonic mean of precision and recall. Calculated as 2 * (precision * recall) / (precision + recall). Reported in percent.
• ACC — range: percent
  • Accuracy, the proportion of correctly classified samples (both benign and malicious). Reported in percent.
• PRC — range: percent
  • Precision, the proportion of predicted malicious flows that are actually malicious. Reported in percent.
• RCL — range: percent
  • Recall, the proportion of actual malicious flows correctly identified. Reported in percent.
• FPR — range: percent
  • False Positive Rate, the proportion of benign flows incorrectly classified as malicious. Reported in percent.

Input / output format

Input: Network traffic flows represented by 24 time-related features.

Output: Binary classification label: malicious or benign.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    prc = tp / (tp + fp) * 100 if (tp + fp) > 0 else 0
    rcl = tp / (tp + fn) * 100 if (tp + fn) > 0 else 0
    f1s = 2 * prc * rcl / (prc + rcl) if (prc + rcl) > 0 else 0
    acc = (tp + tn) / (tp + tn + fp + fn) * 100
    fpr = fp / (fp + tn) * 100 if (fp + tn) > 0 else 0
    return {'ACC': acc, 'PRC': prc, 'RCL': rcl, 'F1S': f1s, 'FPR': fpr}

Common pitfalls

• Adversarial perturbations targeting time-series features cause near-total collapse of precision/recall for regularly trained models.
• Adversarial training improves robustness on the training distribution but may fail to generalize to newer, more complex attacks (e.g., HIKARI dataset).
• False Positive Rate (FPR) must be kept extremely low (<0.4%) to avoid costly enterprise false alarms, which is a key evaluation criterion alongside F1S.

Evidence (verbatim from paper)

The evaluation considers the regular holdout set of the CICIDS2017, NewCICIDS, and HIKARI datasets, as well as the model-specific adversarial holdout sets. ... The ACC, PRC, RCL, F1S, and FPR columns correspond to accuracy, precision, recall, F1-score, and false positive rate.

Citation

@misc{vitorino2024adversarial,
  title={An Adversarial Robustness Benchmark for Enterprise Network Intrusion Detection},
  author={Vitorino et al. (2024)},
  year={2024},
  note={arXiv:2402.16912}
}

• arXiv: 2402.16912