Hogzilla Dataset Eval

hogzilla-dataset-eval

Novel Approach to Intrusion Detection: Introducing GAN-MSCNN-BILSTM with LIME Predictions — Benchama et al. (2024) (arXiv:2406.05443, 2024)

What this evaluates

This evaluation probes a model's capability to detect network intrusions by classifying traffic flows as benign or malicious. It assesses the system's ability to learn complex temporal and multi-scale features from network flow data to distinguish between normal activities and various attack types.

Datasets

• Hogzilla Dataset — total 12832; splits: full (12832)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Ratio of correctly identified instances to the total number of instances.
• Precision — range: [0, 1]
  • Proportion of true positive predictions relative to all positive predictions (TP / (TP + FP)).
• Recall — range: [0, 1]
  • Ratio of true positive predictions to all actual positive instances (TP / (TP + FN)).
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (Precision * Recall) / (Precision + Recall).
• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring discrimination ability across thresholds.

Input / output format

Input: 192 numeric behavioral features per network flow, preprocessed via standard scaling and Pearson correlation filtering (>0.5 with label).

Output: Multi-class label (0: Acceptable, 1: Unrated, 2: Unsafe) or binary label (0: normal, 1: abnormal).

Scoring recipe

def compute_metrics(y_true, y_pred, task='multi'):
    if task == 'binary':
        y_true = (y_true != 0).astype(int)
        y_pred = (y_pred != 0).astype(int)
    tp = np.sum((y_true == 1) & (y_pred == 1))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    accuracy = (tp + tn) / (tp + tn + fp + fn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    return accuracy, precision, recall, f1

Common pitfalls

• The dataset lacks a standard train/val/test split; the paper evaluates on the full 12,832 instances without specifying cross-validation or hold-out procedures.
• Preprocessing involves ad-hoc feature selection based on Pearson correlation > 0.5 with the label, which may leak information if not strictly applied within cross-validation folds.
• Binary vs. multi-class evaluation protocols are reported separately but the exact thresholding or probability aggregation method for ROC-AUC is not detailed.

Evidence (verbatim from paper)

Evaluation metrics are essential tools for assessing the performance of deep learning models, offering valuable insights into their effectiveness and facilitating comparisons between different approaches. In the context of our GAN-MSCNN-BiLSTM framework for intrusion detection, several key evaluation metrics are utilized: Accuracy: measures the ratio of correctly identified instances to the total number of instances, indicating the overall correctness of the model's predictions. Precision: assesses the proportion of true positive predictions relative to all positive predictions, highlighting the model's ability to accurately classify positive instances. Recall (Sensitivity): determines the ratio of true positive predictions to all actual positive instances, showcasing the model's capability to capture relevant positive instances. F1 Score: represents the harmonic mean of precision and recall, offering a balanced assessment of the model's overall performance in binary classification tasks. ROC-AUC: quantifies the area under the Receiver Operating Characteristic (ROC) curve, providing a comprehensive measure of the model's ability to discriminate between intrusion and non-intrusion i

Citation

@misc{benchama2024ganmscnnbilstm,
  title={Novel Approach to Intrusion Detection: Introducing GAN-MSCNN-BILSTM with LIME Predictions},
  author={Benchama et al. (2024)},
  year={2024},
  note={arXiv:2406.05443}
}

• arXiv: 2406.05443