Tii Ssrc 23 Eval

tii-ssrc-23-eval

TII-SSRC-23 Dataset: Typological Exploration of Diverse Traffic Patterns for Intrusion Detection — Herzalla et al. (2023) (arXiv:2310.10661, 2023)

What this evaluates

Evaluates the relative contribution of network traffic features to intrusion detection models by measuring performance degradation when features are shuffled. Probes the model's reliance on specific packet-level and flow-level statistics for distinguishing benign from malicious traffic and classifying attack types.

Datasets

• TII-SSRC-23 — total ?; splits: full (-1)

Metrics

• Permutation Feature Importance (PFI) (primary) — range: other
  • Measures the decrease in a model's performance score when a feature's values are randomly shuffled. A larger drop indicates higher feature importance. Evaluated across multiple classifiers and runs to ensure robustness.

Input / output format

Input: CSV file containing 75 extracted features per bidirectional flow, with labels for 'Label' (Benign/Malicious), 'Traffic Type', and 'Traffic Subtype'.

Output: Ranked feature importance scores, typically aggregated and visualized as boxplots across multiple classifiers and runs.

Scoring recipe

def compute_pfi(model, X, y, score_func):
    base_score = score_func(model.predict(X), y)
    importances = []
    for col in X.columns:
        X_shuffled = X.copy()
        X_shuffled[col] = np.random.permutation(X_shuffled[col])
        shuffled_score = score_func(model.predict(X_shuffled), y)
        importances.append(base_score - shuffled_score)
    return importances

Common pitfalls

• PFI scores can be inflated for correlated features, as shuffling one feature may not fully remove predictive power if correlated features remain.
• Importance rankings vary significantly across different classifier architectures, so relying on a single model yields biased results.
• Multiple random shuffles and runs are required to stabilize estimates, as a single execution can produce noisy importance scores.

Evidence (verbatim from paper)

We employed Permutation Feature Importance (PFI) to compute the feature importance. PFI works by randomly shuffling the values of one feature at a time and then evaluating the resultant effect on the model’s performance. A marked decrease in the model’s performance implies the shuffled feature’s importance for the predictive task in question. However, evaluating feature importance should not entirely depend on a singular execution of PFI. It is advisable to perform multiple runs per method and utilize various classifiers when assessing feature importance. ... We employed three classifiers to calculate feature importance: the Random Forest (RF) classifier, the eXtreme Gradient Boosting (XGBoost) classifier, and the Extra Trees (ET) classifier. ... for each classifier, we conducted three separate runs of PFI

Citation

@misc{herzalla2023tiissrc23,
  title={TII-SSRC-23 Dataset: Typological Exploration of Diverse Traffic Patterns for Intrusion Detection},
  author={Herzalla et al. (2023)},
  year={2023},
  note={arXiv:2310.10661}
}

• arXiv: 2310.10661