Ids Ensemble Eval

ids-ensemble-eval

A Comprehensive Comparative Study of Individual ML Models and Ensemble Strategies for Network Intrusion Detection Systems — Bibers et al. (2024) (arXiv:2410.15597, 2024)

What this evaluates

Evaluates the ability of individual machine learning classifiers and ensemble strategies to detect network intrusions and classify traffic types. It probes model robustness, precision-recall trade-offs, and computational efficiency across diverse real-world network traffic datasets with varying attack profiles.

Datasets

• RoEduNet-SIMARGL2021 — total 30000000; splits: full (30000000)
• CICIDS-2017 — total ?; splits: full (-1)

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * TP / (2 * TP + FP + FN). (Note: The paper contains a typographical error in the denominator, listing 2TN instead of 2TP).
• Accuracy — range: [0, 1]
  • Proportion of correctly classified instances: (TP + TN) / Total.
• Precision — range: [0, 1]
  • Proportion of true positives among all positive predictions: TP / (FP + TP).
• Recall — range: [0, 1]
  • Proportion of true positives among all actual positives: TP / (FN + TP).
• Runtime — range: other
  • Wall-clock time in seconds required for both model training and inference/testing phases.

Input / output format

Input: Tabular network flow features derived from real-time traffic analysis (Netflow-like schema), including numerical and categorical attributes representing packet statistics, protocol types, and connection durations.

Output: Discrete class label indicating traffic type (e.g., Normal, Denial of Service, Malware, Port Scanning, Web Attack, etc.).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p and t == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == p and t == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * tp / (2 * tp + fp + fn) if (2 * tp + fp + fn) > 0 else 0
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1}

Common pitfalls

• The paper defines F1-Score with a typo in the denominator (2TN instead of 2TP), which would yield incorrect values if implemented literally.
• Runtime measurements combine training and testing phases, unlike standard benchmarks that report inference-only latency.
• Several ensemble models (Blending, Stacking, XGBoost, GB) were evaluated on only 20% of the RoEduNet-SIMARGL2021 dataset due to memory constraints, potentially skewing performance comparisons.

Evidence (verbatim from paper)

In this study, the utilization of well-established evaluation metrics is crucial to ascertain the most effective model for integration within an Intrusion Detection System (IDS). Accuracy, precision, recall, and F1-score stand as quintessential performance evaluation metrics. These metrics are derived from four fundamental measures: true positive (TP), false positive (FP), true negative (TN), and false negative (FN) rates. The evaluation metrics are delineated as follows: Accuracy [(TP+TN)/Total]: Signifies the proportion of accurately identified network traffic instances over the total data instances. Precision [TP/(FP+TP)]: Measures the frequency with which the model accurately discerns an attack. Recall [TP/(FN+TP)]: Measures the model’s ability to correctly identify attacks (or intrusions). F1-Score [2TP/(2TN+FP+FN)]: Represents the harmonic mean of precision and recall.

Citation

@misc{bibers2024comprehensive,
  title={A Comprehensive Comparative Study of Individual ML Models and Ensemble Strategies for Network Intrusion Detection Systems},
  author={Bibers et al. (2024)},
  year={2024},
  note={arXiv:2410.15597}
}

• arXiv: 2410.15597