Bs Gat Nf Iot Eval

bs-gat-nf-iot-eval

BS-GAT Behavior Similarity Based Graph Attention Network for Network Intrusion Detection — Wang et al. (2023) (arXiv:2304.07226, 2023)

What this evaluates

Evaluates network intrusion detection capabilities in IoT environments by classifying network flows into benign or multiple attack categories using graph-based and traditional machine learning models.

Datasets

• NF-BoT-IoT-v2 — total 2018770; splits: train (-1), val (-1), test (-1)
• NF-ToN-IoT-v2 — total 934576; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Calculated as (TP + TN) / (TP + FP + FN + TN), where TP, TN, FP, FN are True/False Positives/Negatives from the confusion matrix.
• Precision — range: [0, 1]
  • Calculated as TP / (TP + FP), measuring the proportion of positive identifications that were actually correct.
• Recall — range: [0, 1]
  • Calculated as TP / (TP + FN), measuring the proportion of actual positives that were correctly identified.
• F1-Score — range: [0, 1]
  • Calculated as 2 × (Recall × Precision) / (Recall + Precision), the harmonic mean of Precision and Recall.

Input / output format

Input: 43-dimensional feature vector representing a network flow/node.

Output: Multi-class label from {Benign, Reconnaissance, DDos, Dos, Theft, Backdoor, Injection, MITM, Password, Scanning, XSS}.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p and t == 1)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == p and t == 0)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != p and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t != p and t == 1)
    accuracy = (tp + tn) / (tp + fp + fn + tn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0
    return {'Accuracy': accuracy, 'Precision': precision, 'Recall': recall, 'F1-Score': f1}

Common pitfalls

• The datasets are heavily subsampled (only 5% of majority classes used) to manage memory, which alters the original class distribution and may inflate performance metrics.
• The Gini coefficient evaluates graph construction quality (neighbor uniformity), not model prediction accuracy.
• Baseline hyperparameters are fixed to original paper values rather than tuned on the shared validation set, potentially creating an unfair comparison.

Evidence (verbatim from paper)

To evaluate the performance of the proposed algorithm, the evaluation metrics shown in Table 3 were used. Where TP, FP, FN, and TN represent True Positive, False Positive, True Negative, and False Negative, respectively, in the Confusion Matrix. Table 3: Evaluation metrics adopted in this paper | Metric | Definition | Recall | TP/TP + FN | Precision | TP/TP + FP | F1-Score | 2 × Recall × Precision/Recall + Precision | Accuracy | TP + TN/TP + FP + FN + TN |

Citation

@misc{wang2023bsgat,
  title={BS-GAT Behavior Similarity Based Graph Attention Network for Network Intrusion Detection},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2304.07226}
}

• arXiv: 2304.07226