Pe Malware Classification Eval

pe-malware-classification-eval

A Comprehensive Study on Learning-Based PE Malware Family Classification Methods — Ma et al. (2021) (arXiv:2110.15552, 2021)

What this evaluates

Evaluates learning-based models for PE malware family classification across image, binary, and disassembly input formats. It measures classification accuracy and probes model robustness under concept drift, alongside computational resource overhead.

Datasets

• BIG-15 — total ?; splits: 10-fold CV (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification
• Malimg — total ?; splits: 10-fold CV (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification
• MalwareBazaar — total ?; splits: 10-fold CV (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification
• MalwareDrift — total ?; splits: pre-drift (-1), post-drift (-1); repo https://github.com/MHunt-er/Benchmarking-Malware-Family-Classification

Metrics

• Accuracy (A) — range: percent
  • Percentage of correctly classified samples out of the total.
• Macro Precision ($P_{macro}$) — range: percent
  • Average precision calculated independently for each class and then unweighted mean across all classes.
• Macro Recall ($R_{macro}$) — range: percent
  • Average recall calculated independently for each class and then unweighted mean across all classes.
• Macro F1-score ($F1_{macro}$) (primary) — range: percent
  • Harmonic mean of macro precision and macro recall. Computed as 2 * (P_macro * R_macro) / (P_macro + R_macro).

Input / output format

Input: PE malware files converted to one of three representations: grayscale images, raw binary byte sequences, or disassembly opcode sequences.

Output: Predicted malware family label (multi-class classification).

Scoring recipe

def compute_metrics(y_true, y_pred, num_classes):
    accuracy = sum(1 for t, p in zip(y_true, y_pred) if t == p) / len(y_true)
    precisions, recalls = [], []
    for c in range(num_classes):
        tp = sum(1 for t, p in zip(y_true, y_pred) if t == c and p == c)
        fp = sum(1 for t, p in zip(y_true, y_pred) if t != c and p == c)
        fn = sum(1 for t, p in zip(y_true, y_pred) if t == c and p != c)
        precisions.append(tp / (tp + fp) if (tp + fp) > 0 else 0.0)
        recalls.append(tp / (tp + fn) if (tp + fn) > 0 else 0.0)
    p_macro = sum(precisions) / num_classes
    r_macro = sum(recalls) / num_classes
    f1_macro = 2 * (p_macro * r_macro) / (p_macro + r_macro) if (p_macro + r_macro) > 0 else 0.0
    return accuracy * 100, p_macro * 100, r_macro * 100, f1_macro * 100

Common pitfalls

• Imbalanced class distributions heavily penalize large models (e.g., VGG-16 on BIG-15 where a minority class has <0.4% of samples).
• Evaluating only on static datasets ignores concept drift, which causes average F1-score drops of over 30% in real-world scenarios.
• GPU memory constraints force small batch sizes for large disassembly files, artificially degrading performance for memory-heavy methods like MAGIC.

Evidence (verbatim from paper)

From Table 7, we can observe that all existing methods suffer from a large performance drop while confronting the concept drift in real industry scenarios, where the reduction of F1-score is up to 27.07%-69.62%. The F1-scores for all methods on the post-drift dataset are no more than 45%, reflecting that existing methods fail to consider the scenario of concept drift and there is still much improvement space.

Citation

@misc{ma2021comprehensive,
  title={A Comprehensive Study on Learning-Based PE Malware Family Classification Methods},
  author={Ma et al. (2021)},
  year={2021},
  note={arXiv:2110.15552}
}

• arXiv: 2110.15552