Ugr16 Eval

ugr16-eval

Quality In / Quality Out: Data quality more relevant than model choice in anomaly detection with the UGR'16 — Camacho et al. (2023) (arXiv:2305.19770, 2023)

What this evaluates

Evaluates how data preprocessing choices—such as observation selection, flow directionality, and feature engineering—affect the performance of unsupervised anomaly detection models on network traffic.

Datasets

• UGR'16 — total ?; splits: train (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen positive instance (attack) is ranked higher than a randomly chosen negative instance (normal traffic) across all classification thresholds.

Input / output format

Input: Network traffic observations represented as feature vectors derived from unidirectional or bidirectional packet flows (e.g., port counts, traffic volumes).

Output: Binary anomaly labels or continuous anomaly scores per observation, aggregated into ROC curves and AUC scores per attack type.

Scoring recipe

def compute_auc(y_true, y_scores):
    # y_true: 1 for attack, 0 for normal
    # y_scores: model anomaly scores
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

Common pitfalls

• Including anomalous background traffic (e.g., suspicious MIRC activity in June) in the training set causes unsupervised models to learn it as normal, drastically reducing detection ability for that attack type.
• Flow directionality (unidirectional vs. bidirectional) fundamentally alters feature distributions; unidirectional flows often yield higher AUC for certain attacks (e.g., DOS) while bidirectional flows may be better for others (e.g., NERISBOTNET).

Evidence (verbatim from paper)

Fig. 1(b) represents the AUCs per attack type. Performance differences between the two anomaly detectors are minor in all cases. However, there is a huge difference with respect to including June in the training data (UGR’16v1) or not including it (UGR’16v2).

Citation

@misc{camacho2023qualityin,
  title={Quality In / Quality Out: Data quality more relevant than model choice in anomaly detection with the UGR'16},
  author={Camacho et al. (2023)},
  year={2023},
  note={arXiv:2305.19770}
}

• arXiv: 2305.19770