Ids Smart Grid Eval

ids-smart-grid-eval

Benchmark Evaluation of Anomaly-Based Intrusion Detection Systems in the Context of Smart Grids — Ömer Sen et al. (2023) (arXiv:2312.13705, 2023)

What this evaluates

This benchmark evaluates machine learning-based anomaly detection systems for smart grid cybersecurity, focusing on both detection performance and model explainability. It probes how well intrusion detection methods generalize across diverse operational datasets while providing interpretable feature importance and robustness to data noise.

Datasets

• Power System dataset — total ?; splits: test (-1)
• CIDDS-002 dataset — total ?; splits: test (-1)

Metrics

• AUPC (Area Under Perturbation Curve) — range: [0, 1]
  • Area under the precision-recall curve computed over the top-k most important features (k=5 in the case study). It measures the stability of feature relevance rankings under perturbation.
• explanation sensitivity (Expl.Sens.) (primary) — range: other
  • Quantifies the robustness of feature importance rankings when input data is perturbed or noisy. Higher values indicate more stable and reliable explanations.
• explanation error — range: other
  • Measures the deviation of the model's generated explanation from a ground-truth or baseline explanation.
• explanation time — range: other
  • Computational time required to generate the model explanation.

Input / output format

Input: Normalized tabular or network traffic data conforming to the sklearn.base.BaseEstimator interface, representing smart grid operational scenarios and attack patterns.

Output: Binary anomaly/normal predictions per instance, accompanied by feature importance vectors or explanation scores for interpretability.

Scoring recipe

def evaluate(model, X_test, y_test):
    predictions = model.predict(X_test)
    explanations = model.explain(X_test)
    # Classical detection metrics (implied)
    acc = accuracy_score(y_test, predictions)
    # Explainability metrics
    aupc = area_under_precision_recall_curve(explanations, top_k=5)
    expl_sens = compute_sensitivity(explanations, perturbed_X)
    expl_error = compute_error(explanations, baseline_explanations)
    expl_time = time_taken_to_generate(explanations)
    return {'accuracy': acc, 'AUPC': aupc, 'Expl.Sens.': expl_sens, 'Expl.Error': expl_error, 'Expl.Time': expl_time}

Common pitfalls

• Evaluating only on a single dataset may not capture domain-specific performance variations across different smart grid operational scenarios.
• Ignoring explainability metrics (e.g., AUPC, explanation sensitivity) in favor of classical detection accuracy can obscure model robustness and trustworthiness in industrial control systems.
• Failing to normalize input data consistently before evaluation may skew feature importance and explanation stability results.

Evidence (verbatim from paper)

Regarding the explainability metrics, we observed the following: There was a small difference in explanation error (Cohen’s d = -0.14). No difference was observed in the area under the precision-recall curve (AUPC (Area Under Perturbation Curve)), which means that at least the first five features have the same order of relevance for both approaches (as we chose to evaluate five features for AUPC (Area Under Perturbation Curve)). The most significant effects were seen in explanation time (Cohen’s d = -0.35) and explanation sensitivity (Cohen’s d = 0.31).

Citation

@misc{sen2023benchmark,
  title={Benchmark Evaluation of Anomaly-Based Intrusion Detection Systems in the Context of Smart Grids},
  author={Ömer Sen et al. (2023)},
  year={2023},
  note={arXiv:2312.13705}
}

• arXiv: 2312.13705