Mthl Network Traffic Eval

mthl-network-traffic-eval

Multi-Task Hierarchical Learning Based Network Traffic Analytics — Barut et al. (2021) (arXiv:2106.03850, 2021)

What this evaluates

Evaluates machine learning models on hierarchical network traffic classification tasks, including top-level protocol identification and malware detection, as well as mid-level application and malware type classification.

Datasets

• VPN-nonVPN + $(\mathsf{Net})^2$ + CICIDS2017 — total ?; splits: train (-1), validation (-1); repo https://github.com/ACANETS/NetML-Competition2020

Metrics

• Macro-average F1 score (primary) — range: [0, 1]
  • Computes the unweighted mean of the F1 score across all classes, providing a general insight for multi-class classification problems with imbalanced data.

Input / output format

Input: Metadata features extracted from network flows (100+ engineered flow features).

Output: Hierarchical classification labels: top-level (protocol identification, malware detection) and mid-level (application classification, malware type classification).

Scoring recipe

def macro_f1(y_true, y_pred):
    classes = sorted(set(y_true) | set(y_pred))
    f1s = []
    for c in classes:
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1s.append(f1)
    return sum(f1s) / len(f1s)

Common pitfalls

• The 20% split is explicitly labeled as a 'validation set' in the text but is used as the final evaluation set for baseline comparison.
• Only metadata features are used for all models, explicitly excluding protocol-specific features to expedite efficiency, which may limit performance compared to full-feature baselines.
• Macro-average F1 is chosen specifically to address class imbalance, meaning standard accuracy or micro-F1 would yield misleadingly high scores.

Evidence (verbatim from paper)

Macro-average F1 score is utilized to evaluate the performance since it provides more general insight for multi-class classification problems with imbalanced data. 80% of the training split is used to train the classifiers while the remaining 20% is reserved to for evaluation and referred as validation set.

Citation

@misc{barut2021mthl,
  title={Multi-Task Hierarchical Learning Based Network Traffic Analytics},
  author={Barut et al. (2021)},
  year={2021},
  note={arXiv:2106.03850}
}

• arXiv: 2106.03850