qhjqhj00 ids-automl-eval
Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detection — Li Yang et al. (2024) (arXiv:2409.03141, 2024)
What this evaluates
Evaluates an AutoML-based intrusion detection system's ability to classify network traffic as benign or malicious across multiple attack types. It probes the framework's robustness to class imbalance and its efficiency in real-time network environments.
Datasets
- CICIDS2017 — total ?; splits: test (-1)
- 5G-NIDD — total ?; splits: test (-1)
Metrics
accuracy— range: percent- Proportion of correctly classified samples: (TP + TN) / (TP + TN + FP + FN).
precision— range: percent- Proportion of positive predictions that are correct: TP / (TP + FP).
recall— range: percent- Proportion of actual positives correctly identified: TP / (TP + FN).
F1-score(primary) — range: percent- Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).
training time— range: other- Wall-clock time required to train the final ensemble model.
average test time per sample— range: other- Total inference time divided by the number of test samples.
Input / output format
Input: Tabular network traffic features extracted from captured packets.
Output: Classification label indicating whether traffic is benign or a specific attack type (e.g., DoS, botnet, brute force, infiltration, port scan, web attacks).
Scoring recipe
def compute_metrics(y_true, y_pred):
tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
accuracy = (tp + tn) / (tp + tn + fp + fn)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
return accuracy, precision, recall, f1
Common pitfalls
- Datasets are relatively simple, allowing many baselines to exceed 99% accuracy, which may overstate real-world gains.
- Class imbalance is addressed via synthetic data balancing (TVAE) before training, but metrics are reported on the original test distribution.
- Training time is reported but inference time per sample is emphasized as more critical for edge deployment.
Evidence (verbatim from paper)
To evaluate the proposed AutoML-based IDS framework, two public benchmark network traffic datasets, namely CICIDS2017 and 5G-NIDD, are utilized in the experiments. ... Due to the inherent class imbalance issues in network intrusion detection datasets, four model performance metrics—accuracy, precision, recall, and F1-scores—are considered collectively in the experiments. The F1-score is utilized as the primary performance metric in the performance-based automated model selection and tuning process of the proposed AutoML framework, as it offers a balanced view of anomaly detection results by calculating the harmonic mean of recall and precision. Additionally, the model execution time, involving the training and inference time of the final OCSE model, is utilized to assess the model’s efficiency.
Citation
@misc{yang2024towards,
title={Towards Autonomous Cybersecurity: An Intelligent AutoML Framework for Autonomous Intrusion Detection},
author={Li Yang et al. (2024)},
year={2024},
note={arXiv:2409.03141}
}
- arXiv: 2409.03141