Xai Feature Selection Ids Eval

xai-feature-selection-ids-eval

XAI-based Feature Selection for Improved Network Intrusion Detection Systems — Arreche et al. (2024) (arXiv:2410.10050, 2024)

What this evaluates

This evaluation probes the effectiveness of explainable AI (XAI)-driven feature selection methods on network intrusion detection systems. It measures how well various black-box machine learning models classify network traffic flows into normal or specific attack categories when trained on different subsets of extracted features.

Datasets

• CICIDS-2017 — total 2775364; splits: full (2775364)
• RoEduNet-SIMARGL2021 — total 31433875; splits: full (31433875)

Metrics

• Accuracy (Acc) (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• F1-score (F1) — range: [0, 1]
  • Harmonic mean of precision and recall, typically macro-averaged across classes.
• Balanced Accuracy (Bacc) — range: [0, 1]
  • Average of recall obtained on each class, accounting for class imbalance.
• Matthews Correlation Coefficient (MCC) — range: [-1, 1]
  • Correlation coefficient between observed and predicted binary classifications, robust to imbalance.
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring trade-off between true positive and false positive rates.

Input / output format

Input: Tabular network flow features including packet lengths, TCP flags, port numbers, flow durations, and inter-arrival times.

Output: Categorical label indicating traffic type: Normal, DoS, PortScan, Brute Force, Web Attack, Bot, or Infiltration.

Scoring recipe

import numpy as np
from sklearn.metrics import accuracy_score, f1_score, roc_auc_score

def compute_metrics(y_true, y_pred, y_prob=None):
    acc = accuracy_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred, average='macro')
    auc = roc_auc_score(y_true, y_prob, multi_class='ovr') if y_prob is not None else None
    return {'Acc': acc, 'F1': f1, 'AUC-ROC': auc}

Common pitfalls

• The datasets are severely imbalanced (e.g., CICIDS-2017 is 84.4% Normal traffic), making Accuracy alone misleading without macro-averaged F1 or Balanced Accuracy.
• The paper uses a custom weighted ranking system (3/2/1 points for top-3 methods across k=5, 10, 15 feature subsets) to compare feature selection techniques, which is non-standard and lacks a publicly available implementation.
• No explicit train/validation/test split is defined in the text, preventing exact reproduction of the reported performance numbers.

Evidence (verbatim from paper)

Black-box AI Metrics: We use a commonly used set of criteria for assessing intrusion detection and classification issues to evaluate the performance of the black-box AI models. The measures that fall under this category include balanced accuracy (Bacc), Matthews correlation coefficient (MCC), accuracy (Acc), precision (Prec), recall (Rec), and F1-score (F1). The effectiveness of the AI model is also determined by calculating the AucRoc (area under the ROC curve) score.

Citation

@misc{arreche2024xai,
  title={XAI-based Feature Selection for Improved Network Intrusion Detection Systems},
  author={Arreche et al. (2024)},
  year={2024},
  note={arXiv:2410.10050}
}

• arXiv: 2410.10050