Netflow Nids Eval

netflow-nids-eval

Towards a Standard Feature Set for Network Intrusion Detection System Datasets — Sarhan et al. (2021) (arXiv:2101.11315, 2021)

What this evaluates

Evaluates the generalizability and detection performance of machine learning classifiers for network intrusion detection when using a standardized NetFlow feature set across multiple benchmark datasets. It probes whether a common feature representation improves cross-dataset model accuracy and reduces false alarms compared to proprietary or basic NetFlow features.

Datasets

• NF-UNSW-NB15-v2 — total ?; splits: train (-1), test (-1)
• NF-BoT-IoT-v2 — total ?; splits: train (-1), test (-1)
• NF-ToN-IoT-v2 — total ?; splits: train (-1), test (-1)
• NF-CSE-CIC-IDS2018-v2 — total ?; splits: train (-1), test (-1)
• NF-UQ-NIDS-v2 — total ?; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified samples out of the total number of samples.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the model's ability to distinguish between classes across all classification thresholds.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall. For multi-class, the paper reports the weighted average.
• Detection Rate (DR) — range: [0, 1]
  • Recall or true positive rate for attack classes.
• False Alarm Rate (FAR) — range: [0, 1]
  • False positive rate, proportion of benign samples incorrectly classified as attacks.
• Prediction Time (µs) — range: other
  • Average time in microseconds required to predict a single test sample.

Input / output format

Input: Vector of standardized NetFlow features (12 or 43 dimensions) extracted from network flows. Flow identifiers (IDs, IPs, ports, timestamps) are removed. Features are min-max normalized to [0, 1].

Output: Class label: either binary (Benign/Attack) or multi-class (specific attack types such as DoS, DDoS, Brute Force, etc.).

Scoring recipe

def compute_metrics(y_true, y_pred, y_proba=None):
    accuracy = (y_true == y_pred).mean()
    tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel()
    dr = tp / (tp + fn) if (tp + fn) > 0 else 0
    far = fp / (fp + tn) if (fp + tn) > 0 else 0
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    f1 = 2 * precision * dr / (precision + dr) if (precision + dr) > 0 else 0
    auc = roc_auc_score(y_true, y_proba) if y_proba is not None else 0
    return accuracy, auc, f1, dr, far

Common pitfalls

• Flow identifiers (IPs, ports, timestamps) must be dropped before training to prevent the model from learning dataset-specific node biases rather than attack patterns.
• TTL-based features are explicitly dropped for UNSW-NB15 due to extreme correlation with labels, which can cause data leakage or overfitting.
• Results are averaged over five cross-validation splits; reporting a single split will not match the paper's numbers.
• Min-max normalization to [0, 1] is applied to all features; skipping this will degrade performance and mismatch reported metrics.

Evidence (verbatim from paper)

The evaluation is conducted by comparing the classifier performance with the corresponding metrics of the basic NetFlow and original datasets. Various classification metrics are collected such as accuracy, Area Under the Curve (AUC), F1 Score, Detection Rate (DR), False Alarm Rate (FAR) and time required to predict a single test sample in microseconds (µs). ... The datasets have been split into 70%-30% for training and testing purposes. For a fair evaluation, five cross-validation splits are conducted and the mean is measured.

Citation

@misc{sarhan2021netflow,
  title={Towards a Standard Feature Set for Network Intrusion Detection System Datasets},
  author={Sarhan et al. (2021)},
  year={2021},
  note={arXiv:2101.11315}
}

• arXiv: 2101.11315