Iot Intrusion Detection Eval

iot-intrusion-detection-eval

EG-ConMix: An Intrusion Detection Method based on Graph Contrastive Learning — Lijin Wu et al. (arXiv:2403.17980, 2024)

What this evaluates

This evaluation probes an intrusion detection model's ability to classify network traffic flows as benign or malicious across highly imbalanced IoT datasets. It specifically tests the model's robustness to extreme class imbalance and its capacity to leverage graph-structured representations of network flows for anomaly detection.

Datasets

• BoT-IoT — total 3668522; splits: train (-1), val (-1), test (-1)
• ToN-IoT — total 22339021; splits: train (-1), val (-1), test (-1)

Metrics

• macro-F1 (primary) — range: [0, 1]
  • F1-score = 2 × (Recall × Precision) / (Recall + Precision). Macro-F1 is computed by calculating the F1 score for each class independently and then averaging them across all categories.

Input / output format

Input: Graph-structured network traffic data where nodes and edges represent network entities/flows, with each instance containing node/edge features (e.g., 55 features for BoT-IoT, 30 for ToN-IoT) and topological relationships.

Output: Multi-class classification label indicating whether a network flow/edge is benign or belongs to a specific attack category.

Scoring recipe

def compute_macro_f1(y_true, y_pred, num_classes):
    f1_scores = []
    for c in range(num_classes):
        tp = sum((y_true == c) & (y_pred == c))
        fp = sum((y_true != c) & (y_pred == c))
        fn = sum((y_true == c) & (y_pred != c))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        f1_scores.append(f1)
    return sum(f1_scores) / len(f1_scores)

Common pitfalls

• Extreme class imbalance (e.g., 0.01% benign in BoT-IoT) makes accuracy misleading; macro-F1 must be used to fairly evaluate minority classes.
• The paper fixes the positive/negative sample ratio across train/val/test splits, deviating from standard random splitting protocols.
• Evaluation uses a 70/10/20 split but also reports results from 10-fold cross-validation repeated 5 times; failing to replicate the repeated CV may cause result mismatches.

Evidence (verbatim from paper)

The overall accuracy performance of the model is measured by the F1 score, a metric that calculates the reconciled mean of sensitivity and accuracy, and is a widely used evaluation metric for models trained using unbalanced datasets, denoted as follows: F1-score=2×(Recall×Precision)/(Recall+Precision)... In order to clearly distinguish the differences and benefits of our model in anomaly detection, we compared the macro F1 scores between several different algorithms. When calculating macro F1 scores, the F1 scores of all categories are averaged.

Citation

@misc{wu2024egconmix,
  title={EG-ConMix: An Intrusion Detection Method based on Graph Contrastive Learning},
  author={Lijin Wu et al.},
  year={2024},
  note={arXiv:2403.17980}
}

• arXiv: 2403.17980