Flowtransformer Eval

flowtransformer-eval

FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems — Manocchio et al. (2023) (arXiv:2304.14746, 2023)

What this evaluates

This evaluation protocol assesses the effectiveness of various transformer-based architectures for flow-based network intrusion detection. It systematically tests different input encodings, transformer blocks, and classification heads across three standard NIDS datasets to determine optimal configurations for accuracy, model size, and inference speed.

Datasets

• NSL-KDD — total ?; splits: unspecified (-1)
• UNSW-NB15 — total ?; splits: unspecified (-1)
• CSE-CIC-IDS2018 — total ?; splits: unspecified (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall, computed from True Positives (TP), False Positives (FP), and False Negatives (FN).
• false alarm rate — range: [0, 1]
  • Ratio of False Positives to all actual negatives, computed as FP / (FP + TN).
• detection rate — range: [0, 1]
  • Ratio of True Positives to all actual positives, computed as TP / (TP + FN).

Input / output format

Input: Flow-level network traffic features converted to a standardized flow format, processed through configurable input encodings.

Output: Predicted class labels (normal traffic or specific attack categories) generated via a configurable classification head.

Scoring recipe

def compute_metrics(tp, tn, fp, fn):
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    far = fp / (fp + tn) if (fp + tn) > 0 else 0.0
    dr = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return {'f1': f1, 'false_alarm_rate': far, 'detection_rate': dr}

Common pitfalls

• Using Global Average Pooling as the classification head performs poorly for NIDS despite its dominance in text NLP.
• Failing to repeat each grid search configuration at least 3 times, which can skew results due to poor model initialization.
• Measuring inference time without clearing the GPU cache or taking a median over multiple runs, leading to inaccurate throughput estimates.

Evidence (verbatim from paper)

To assess the performance of different transformer models, standard metrics were utilised, such as F1 score, false alarm rate and detection rate. The metrics are computed using a combination of True Positives, True Negatives, False Positives, and False Negatives, denoted as $TP$, $TN$, $FP$, and $FN$ respectively. We use both F1 score as the primary metrics to compare approaches.

Citation

@misc{manocchio2023flowtransformer,
  title={FlowTransformer: A Transformer Framework for Flow-based Network Intrusion Detection Systems},
  author={Manocchio et al. (2023)},
  year={2023},
  note={arXiv:2304.14746}
}

• arXiv: 2304.14746