Cic Ids2017 Nids Eval

cic-ids2017-nids-eval

Unveiling the potential of Graph Neural Networks for robust Intrusion Detection — Pujol-Perich et al. (2021) (arXiv:2107.14756, 2021)

What this evaluates

Evaluates the classification accuracy and adversarial robustness of a Graph Neural Network-based Network Intrusion Detection System (NIDS) on distinguishing benign traffic from various attack types in network flow data.

Datasets

• CIC-IDS2017 — total 1119250; splits: train (895400), val (223850)

Metrics

• weighted F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, weighted by the number of true instances (support) in each class. Computed as the sum of per-class F1-scores multiplied by their relative class frequencies.

Input / output format

Input: Host-connection graphs constructed from network flow records, where each flow aggregates 80 features representing packet sizes, inter-arrival times, and other traffic statistics.

Output: Discrete class label indicating either 'Benign' or one of 11 attack sub-classes (e.g., SSH-Patator, DoS GoldenEye, DDoS, etc.).

Scoring recipe

def compute_weighted_f1(y_true, y_pred, classes):
    precisions, recalls, supports = [], [], []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        sup = sum(1 for t in y_true if t == c)
        precisions.append(prec); recalls.append(rec); supports.append(sup)
    total_sup = sum(supports)
    return sum(f1 * (s / total_sup) for f1, s in zip(precisions, supports))

Common pitfalls

• Dataset is highly imbalanced (~88% benign, ~12% attacks); the authors drop 90% of benign training graphs to over-represent attacks, altering the evaluation distribution compared to raw data.
• Evaluation only includes classes with >100 flow samples, excluding rare attack types from the final metric calculation.
• Adversarial robustness is tested only by perturbing packet size and inter-arrival time, not by modifying the underlying graph structure or other flow features.

Evidence (verbatim from paper)

We use a standard weighted F1-score to measure the per-class accuracy, which unifies in a single metric the precision and recall of solutions. From these results, we can observe that the proposed model achieves a level of accuracy comparable to state-of-the-art ML methods, obtaining a weighted F1-score of 0.99 over all traffic flows.

Citation

@misc{pujolperich2021gnnids,
  title={Unveiling the potential of Graph Neural Networks for robust Intrusion Detection},
  author={Pujol-Perich et al. (2021)},
  year={2021},
  note={arXiv:2107.14756}
}

• arXiv: 2107.14756