Malware Family Classification Eval

malware-family-classification-eval

Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models — Bai et al. (2026) (arXiv:2604.02490, 2026)

What this evaluates

Evaluates the ability of LLMs and their ensembles to correctly classify malware samples into one of ten canonical families based on their behavior or code semantics. It probes robustness to class imbalance and the effectiveness of hierarchical decision-making under obfuscation.

Datasets

• Gold-standard malware family dataset — total 200; splits: test (200)

Metrics

• Accuracy — range: [0, 1]
  • Fraction of correctly classified samples out of the total number of samples.
• Macro Precision — range: [0, 1]
  • Unweighted mean of precision calculated for each of the ten malware families independently.
• Macro Recall — range: [0, 1]
  • Unweighted mean of recall calculated for each of the ten malware families independently.
• Macro F1-score (primary) — range: [0, 1]
  • Harmonic mean of macro precision and macro recall, providing a balanced measure across all classes regardless of imbalance.

Input / output format

Input: Malware samples (code/behavioral representations) to be classified into one of ten canonical malware families.

Output: A single canonical malware family label from the predefined set of ten.

Scoring recipe

def compute_metrics(predictions, gold_labels, num_classes=10):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    precisions, recalls, f1s = [], [], []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
        precisions.append(prec)
        recalls.append(rec)
        f1s.append(f1)
    macro_prec = sum(precisions) / num_classes
    macro_rec = sum(recalls) / num_classes
    macro_f1 = sum(f1s) / num_classes
    return accuracy, macro_prec, macro_rec, macro_f1

Common pitfalls

• The dataset is very small (200 samples), so results lack statistical significance testing and may not generalize.
• Macro-averaging is used to handle class imbalance, which can mask poor performance on minority families.
• Model weights for the ensemble are derived from Macro-F1 scores computed on the same gold-standard dataset, risking overfitting to the evaluation set.

Evidence (verbatim from paper)

Performance was measured using Accuracy, Macro Precision, Macro Recall, and Macro F1-score. Macro-averaged metrics were emphasized due to class imbalance and to ensure equal importance across malware families.

Citation

@misc{bai2026automated,
  title={Automated Malware Family Classification using Weighted Hierarchical Ensembles of Large Language Models},
  author={Bai et al. (2026)},
  year={2026},
  note={arXiv:2604.02490}
}

• arXiv: 2604.02490