Ids Detection Eval

ids-detection-eval

Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction — Talukder et al. (2024) (arXiv:2401.12262, 2024)

What this evaluates

Evaluates machine learning classifiers for network intrusion detection on imbalanced, high-dimensional traffic data. Probes the model's ability to distinguish benign from malicious traffic across binary and multilabel settings using standard classification metrics.

Datasets

• UNSW-NB15 — total ?; splits: train (-1), test (-1)
• CIC-IDS2017 — total ?; splits: train (-1), test (-1)
• CIC-IDS2018 — total ?; splits: train (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted observations to total observations. Formula: (TP + TN) / (TP + FP + FN + TN).
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive values to total predicted positive values. Formula: TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive values to all actual positive values. Formula: TP / (TP + FN).
• F1-Score — range: [0, 1]
  • Harmonic mean of precision and recall. Formula: 2 * (Precision * Recall) / (Precision + Recall).
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between class labels by plotting True Positive Rate against False Positive Rate at various thresholds.

Input / output format

Input: Feature vectors extracted from network traffic records, preprocessed via Random Oversampling, Stacking Feature Embedding, and PCA.

Output: Predicted class label (binary or multilabel classification).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum((t==1) & (p==1) for t, p in zip(y_true, y_pred))
    tn = sum((t==0) & (p==0) for t, p in zip(y_true, y_pred))
    fp = sum((t==0) & (p==1) for t, p in zip(y_true, y_pred))
    fn = sum((t==1) & (p==0) for t, p in zip(y_true, y_pred))
    acc = (tp + tn) / (tp + fp + fn + tn)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * (prec * rec) / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1

Common pitfalls

• Data imbalance is handled via Random Oversampling on the training folds, so test metrics must be computed on the original, unbalanced test split to reflect real-world performance.
• The protocol evaluates both binary and multilabel classification; metric aggregation (e.g., macro vs micro averaging) is not explicitly specified and may vary across implementations.
• 10-fold cross-validation is used with a 90/10 train/test split per fold, meaning reported scores are averages across 10 runs rather than a single holdout evaluation.

Evidence (verbatim from paper)

Several measures, such as accuracy, precision, recall, F1-score, ROC curve, and confusion matrix, are used to evaluate the performance of our proposed model. ... Accuracy is a fundamental performance metric, representing the proportion of correctly predicted observations to the total observations. It is calculated as follows: $$ Accuracy = rac {T P + T N}{T P + F P + F N + T N} $$

Citation

@misc{talukder2024network,
  title={Machine learning-based network intrusion detection for big and imbalanced data using oversampling, stacking feature embedding and feature extraction},
  author={Talukder et al. (2024)},
  year={2024},
  note={arXiv:2401.12262}
}

• arXiv: 2401.12262