Iot Nids Eval

iot-nids-eval

E-GraphSAGE: A Graph Neural Network based Intrusion Detection System for IoT — Lo et al. (2021) (arXiv:2103.16329, 2021)

What this evaluates

Evaluates the ability of a graph neural network to detect network intrusions in IoT environments by classifying traffic flows as benign or malicious, and identifying specific attack types. It probes the model's capacity to leverage topological graph structures and edge features for robust intrusion detection across imbalanced, real-world network traffic datasets.

Datasets

• BoT-IoT — total ?; splits: train (-1), test (-1)
• NF-BoT-IoT — total ?; splits: train (-1), test (-1)
• ToN-IoT — total ?; splits: train (-1), test (-1)
• NF-ToN-IoT — total ?; splits: train (-1), test (-1)

Metrics

• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). For multiclass tasks, the paper reports the Weighted Average F1-Score, which computes the F1 per class and averages them weighted by class support.
• Detection Rate (Recall) — range: [0, 1]
  • True Positives divided by the sum of True Positives and False Negatives: TP / (TP + FN).
• Precision — range: [0, 1]
  • True Positives divided by the sum of True Positives and False Positives: TP / (TP + FP).
• Accuracy — range: [0, 1]
  • Sum of True Positives and True Negatives divided by total samples: (TP + TN) / (TP + FP + TN + FN).
• False Alarm Rate (FAR) — range: [0, 1]
  • False Positives divided by the sum of False Positives and True Negatives: FP / (FP + TN).

Input / output format

Input: Network flow records represented as graphs with edge features (e.g., packet counts, duration) and topological structure.

Output: Classification label: binary (attack vs. benign) or multiclass (specific attack type + benign).

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t != 1 and p != 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != 1 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p != 1)
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    accuracy = (tp + tn) / (tp + fp + tn + fn)
    far = fp / (fp + tn) if (fp + tn) > 0 else 0
    return {"F1-Score": f1, "Recall": recall, "Precision": precision, "Accuracy": accuracy, "FAR": far}

Common pitfalls

• Class imbalance makes Accuracy misleading; the paper explicitly states F1-Score is more relevant and should be used for SOTA comparison.
• NetFlow dataset variants use generic features instead of engineered ones, causing significant performance drops compared to original datasets.
• ToN-IoT evaluation uses only a 10% random subset due to dataset size, which may affect generalization claims.

Evidence (verbatim from paper)

Since the considered datasets are generally highly imbalanced, the F1-Score is a more relevant performance metric. We use the F1-Score to compare our classifier with the state-of-the-art, i.e., the best classification results reported in the literature for each of the four NIDS datasets.

Citation

@misc{lo2021egraphsage,
  title={E-GraphSAGE: A Graph Neural Network based Intrusion Detection System for IoT},
  author={Lo et al. (2021)},
  year={2021},
  note={arXiv:2103.16329}
}

• arXiv: 2103.16329