Paella Malware Detection Eval

paella-malware-detection-eval

pAElla: Edge-AI based Real-Time Malware Detection in Data Centers — Libri et al. (2020) (arXiv:2004.03670, 2020)

What this evaluates

Evaluates machine learning models' ability to detect malware and anomalies in data center compute nodes using high-resolution power consumption data and hardware performance counters. It probes real-time anomaly detection capabilities under severe class imbalance and varying computational workloads.

Datasets

• pAElla Malware & Benchmark Dataset — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/alibriee/paella

Metrics

• weighted F1-score (primary) — range: [0, 1]
  • Accounts for class imbalance: F1 = (2·TP·W_M) / (2·TP·W_M + FP·W_H + FN·W_M), where W_M and W_H are the counts of malware and healthy instances. Also reports False Alarm (FA) rate (FP rate) and Malware Miss (MM) rate (FN rate).

Input / output format

Input: Time-series feature vectors per 20 ms window containing hardware performance counters (per-core and per-cpu), Power Spectral Density (PSD) estimates from high-resolution power measurements, and coarse-grain power statistics (mean, std, max, min).

Output: Binary label: 'malware/anomaly' or 'healthy/normal'.

Scoring recipe

def compute_metrics(tp, fp, fn, tn, W_M, W_H):
    fa_rate = fp / (fp + tn)  # False Alarm rate (FP rate)
    mm_rate = fn / (tp + fn)  # Malware Miss rate (FN rate)
    f1_weighted = (2 * tp * W_M) / (2 * tp * W_M + fp * W_H + fn * W_M)
    return fa_rate, mm_rate, f1_weighted

Common pitfalls

• Dataset is highly imbalanced (~95% malware), so unweighted F1 or accuracy is misleading.
• Threshold selection for anomaly scoring (e.g., reconstruction error or outlier percentage) significantly impacts FA and MM rates.
• Performance counter feature space is often not linearly separable for oc-SVM, requiring careful PCA/kernel tuning.

Evidence (verbatim from paper)

Table III reports the results of our analysis in the test set. Namely, the (i) False Alarm (FA) rate (= False Positive - FP - rate) - i.e., healthy benchmarks erroneously labeled as malware; (ii) the Malware Miss (MM) rate (= False Negative - FN - rate) - i.e., malware not detected; and (iii) the weighted F1-score[28] (best value at 1, and worst at 0), which measure the test accuracy based on the following formula: F1score = (2TP·WM)/(2TP·WM + FP·WH + FN·WM) where we weighted the True Positives (TP), FP, and FN by the number of instances of each class (Malware and Healthy), via the two weights WM and WH, to take into account the imbalance of the dataset between number of malware and number of healthy acquisitions.

Citation

@misc{libri2020paella,
  title={pAElla: Edge-AI based Real-Time Malware Detection in Data Centers},
  author={Libri et al. (2020)},
  year={2020},
  note={arXiv:2004.03670}
}

• arXiv: 2004.03670