Ics Flow Eval

ics-flow-eval

Anomaly Detection Dataset for Industrial Control Systems — Dehlaghi-Ghadim et al. (2023) (arXiv:2305.09678, 2023)

What this evaluates

Evaluates machine learning models' ability to detect and classify cyberattacks in Industrial Control Systems (ICS) network traffic. It probes the capability to distinguish between normal operations and specific attack types like DDoS, IP-Scan, MitM, Port-Scan, and Replay using flow-level features.

Datasets

• ICS-Flow — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Calculated per class and typically macro-averaged to handle imbalanced datasets.
• Accuracy — range: [0, 1]
  • Ratio of correctly classified instances to the total number of instances.

Input / output format

Input: Network flow records or feature vectors extracted from ICS network traffic.

Output: Categorical label indicating the flow type: Normal, DDoS, IP-Scan, MitM, Port-Scan, or Replay.

Scoring recipe

def compute_f1(predictions, gold):
    tp = fp = fn = 0
    for p, g in zip(predictions, gold):
        if p == g == 'positive': tp += 1
        elif p == 'positive' and g != 'positive': fp += 1
        elif p != 'positive' and g == 'positive': fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

Common pitfalls

• Relying solely on accuracy can be misleading due to class imbalance; the paper explicitly notes F1-score is more reliable for imbalanced classes.
• Individual flow analysis cannot differentiate between attacks sharing similar network processes (e.g., IP-Scan vs. Port-Scan/Replay/MitM due to ARP poisoning), leading to high misclassification rates for those specific classes.

Evidence (verbatim from paper)

Despite this, the ML methods effectively identified normal flows, as evidenced by their F1-score exceeding 0.99. The high F1-score for DDoS attack identification was expected since DDoS flows are easily detectable by monitoring massive packets and connections during the attack. Conversely, the F1-scores for IP-scan, Port-Scan, MitM, and Replay attack indicate that accurately classifying flows into the correct attack types is a non-trivial task, especially for IP-Scan with an F1-score of only 0.52.

Citation

@misc{dehlaghighadim2023icsflow,
  title={Anomaly Detection Dataset for Industrial Control Systems},
  author={Dehlaghi-Ghadim et al. (2023)},
  year={2023},
  note={arXiv:2305.09678}
}

• arXiv: 2305.09678