Malware Classification Eval

malware-classification-eval

Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection — Alam et al. (2024) (arXiv:2403.17978, 2024)

What this evaluates

This evaluation probes a model's ability to classify long-sequence binary and executable files into malware families or benign/malicious categories. It tests robustness to varying sequence lengths, compression formats, and real-world malware distribution characteristics compared to synthetic long-range benchmarks.

Datasets

• Kaggle (BIG 2015) — total 10868; splits: train (8694), test (2174)
• Drebin — total 4664; splits: train (3731), test (933)
• EMBER — total 800000; splits: train (600000), test (200000)
• LRA — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances. Reported as mean accuracy with standard deviation over 10-fold cross-validation for Kaggle and Drebin, and single-run accuracy for EMBER and LRA tasks.
• execution_time — range: seconds
  • Wall-clock time per training epoch or inference, measured in seconds. Models exceeding 10,000 seconds per epoch are marked as out-of-time (OOT).

Input / output format

Input: Raw binary executables, assembly files, or APK/TAR archives converted into token sequences. Sequences are padded or truncated to a fixed maximum length (e.g., 256 to 131,072 tokens).

Output: Discrete class labels corresponding to malware families (for Kaggle/Drebin) or binary benign/malicious classification (for EMBER).

Scoring recipe

predictions = model.predict(sequences)
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
accuracy = correct / len(gold_labels)
return accuracy

Common pitfalls

• Sequence length truncation/padding significantly impacts performance; models must handle variable-length binaries up to 100M+ tokens.
• Dataset splits are stratified by class (80-20 per class), not random global split, which affects cross-validation folds.
• LRA benchmark scores are poor proxies for real-world malware detection; high LRA accuracy does not guarantee strong malware classification performance.

Evidence (verbatim from paper)

To evaluate the performance, the proposed HGConv is compared with other state-of-the-art (SOTA) sequence models. For Kaggle and Drebin datasets, the proposed method is compared with non-attention-based processors... [Table 1] shows the mean accuracy with standard deviation for 10-fold cross-validation for each of the methods.

Citation

@misc{alam2024holographic,
  title={Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection},
  author={Alam et al. (2024)},
  year={2024},
  note={arXiv:2403.17978}
}

• arXiv: 2403.17978