Kitsune Iot Nids Eval

kitsune-iot-nids-eval

Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection — Mirsky et al. (2018) (arXiv:1802.09089, 2018)

What this evaluates

Evaluates online unsupervised anomaly detection systems for network intrusion detection on IoT surveillance and network traffic. It measures how well models distinguish between normal traffic and various attack types (e.g., DoS, MITM, malware) using streaming packet features.

Datasets

• Kitsune IoT Network Datasets — total ?; splits: train (1000000), test (-1); repo https://github.com/ymirsky/KitNET-py

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Represents the probability that a classifier ranks a randomly chosen anomalous instance higher than a randomly chosen normal instance.
• EER — range: [0, 1]
  • Equal Error Rate. The value where False Negative Rate (FNR) and False Positive Rate (FPR) are minimal and equal to each other.
• TPR — range: [0, 1]
  • True Positive Rate = TP / (TP + FN). Measured at fixed FPR thresholds of 0 and 0.001.
• FNR — range: [0, 1]
  • False Negative Rate = FN / (FN + TP). Measured at fixed FPR thresholds of 0 and 0.001.

Input / output format

Input: 198 statistical features extracted from network packets processed sequentially in a streaming fashion.

Output: Continuous anomaly score in range [0, ∞). Scores are normalized by a threshold φ; scores < 1 indicate normal traffic, scores > 1 indicate anomalies. Binary classification is derived by applying φ.

Scoring recipe

def compute_auc_eer(scores, labels):
    tprs, fprs = [], []
    for thr in np.linspace(0, max(scores), 1000):
        tp = sum(1 for s, l in zip(scores, labels) if s >= thr and l == 1)
        fp = sum(1 for s, l in zip(scores, labels) if s >= thr and l == 0)
        fn = sum(1 for s, l in zip(scores, labels) if s < thr and l == 1)
        tn = sum(1 for s, l in zip(scores, labels) if s < thr and l == 0)
        tprs.append(tp / (tp + fn) if (tp + fn) > 0 else 0)
        fprs.append(fp / (fp + tn) if (fp + tn) > 0 else 0)
    auc = np.trapz(tprs, fprs)
    eer = min(abs(t - f) for t, f in zip(tprs, fprs))
    return auc, eer

Common pitfalls

• Threshold φ drastically affects FPR/FNR; the paper fixes FPR at 0 or 0.001 when reporting TPR/FNR, so comparing raw thresholds across papers is invalid.
• Online vs offline baselines: Offline models (IF, GMM) train on the full dataset, while Kitsune trains incrementally on the first 1M packets; direct performance comparison requires acknowledging this architectural difference.
• Parameter m controls ensemble size; lower m (e.g., 1) yields higher detection accuracy but slower speed, while higher m (e.g., 10) speeds up processing at the cost of some accuracy.

Evidence (verbatim from paper)

To measure the general performance (i.e. with every possible φ), we used the area under the receiver operating characteristic curve (AUC), and the equal error rate (EER). In our context, the AUC is the probability that a classifier will rank a randomly chosen anomalous instance higher than a randomly chosen normal instance. In other words, an algorithm with an AUC of 1 is a perfect anomaly detector on the given dataset, whereas an algorithm with an AUC of 0.5 is randomly guessing labels. The EER is a measure which captures an algorithm's trade-off between its FNR and FPR. It is computed as the value of FNR and FPR when they are minimal and equal to one another.

Citation

@misc{mirsky2018kitsune,
  title={Kitsune: An Ensemble of Autoencoders for Online Network Intrusion Detection},
  author={Mirsky et al. (2018)},
  year={2018},
  note={arXiv:1802.09089}
}

• arXiv: 1802.09089