Malware Clustering Eval

malware-clustering-eval

Clustering Malware at Scale: A First Full-Benchmark Study — Mocko et al. (2025) (arXiv:2511.23198, 2025)

What this evaluates

This benchmark evaluates the effectiveness of unsupervised clustering algorithms on large-scale malware binary datasets. It probes how well different feature representations and clustering methods can group malware samples into coherent families while handling real-world noise and benign samples.

Datasets

• Bodmas — total ?; splits: train (-1), test (-1)
• Ember — total ?; splits: train (-1), test (-1)
• Security — total ?; splits: train (-1), test (-1)

Metrics

• Homogeneity (primary) — range: percent
  • Measures whether each cluster contains only members of a single class. Calculated as 1 - (H(C) + H(K)) / H(C,K), where H(C) is entropy of cluster assignments, H(K) is entropy of class labels, and H(C,K) is joint entropy. Values are reported as percentages in the paper.
• V-Measure — range: percent
  • Harmonic mean of homogeneity and completeness. Penalizes over-clustering by incorporating completeness, which measures whether all members of a given class are assigned to the same cluster.

Input / output format

Input: Numerical feature representations (embeddings) of malware binaries, generated using PCA, UMAP, or Autoencoder dimensionality reduction.

Output: Cluster assignment labels for each input sample.

Scoring recipe

def compute_homogeneity(predictions, gold):
    # predictions: cluster labels assigned by algorithm
    # gold: true malware family labels
    from sklearn.metrics import homogeneity_score
    return homogeneity_score(gold, predictions) * 100

Common pitfalls

• High Homogeneity can be artificially inflated by creating too many clusters (e.g., one per sample), which is why V-Measure remains low despite high Homogeneity scores.
• Dataset composition heavily influences clustering quality; Ember underperforms Bodmas likely due to underlying malware family distributions rather than the number of clusters or algorithm choice.
• Including benign samples does not degrade clustering performance, contrary to common assumptions in malware analysis.

Evidence (verbatim from paper)

Homogeneity was picked as the primary metric to show us the difference in quality (or lack thereof) in this experiment. Due to time constraints, we trained each representation only once. We also ran K-Means only once on the trained representations.

Citation

@misc{mocko2025clustering,
  title={Clustering Malware at Scale: A First Full-Benchmark Study},
  author={Mocko et al. (2025)},
  year={2025},
  note={arXiv:2511.23198}
}

• arXiv: 2511.23198