Vae Malware Detection Eval

vae-malware-detection-eval

Leveraging VAE-Derived Latent Spaces for Enhanced Malware Detection with Machine Learning Classifiers — Ajayi et al. (2025) (arXiv:2503.20803, 2025)

What this evaluates

Evaluates the effectiveness of Variational Autoencoder (VAE)-derived latent space features for malware classification using traditional machine learning models. It probes robustness to data partitioning, random seed initialization, and computational efficiency without hyperparameter tuning.

Datasets

• EMBER — total ?; splits: 30/30 (30% train, 30% test, 40% holdout) (-1), 50/30 (50% train, 30% test, 20% holdout) (-1), 70/30 (70% train, 30% test) (-1)
• BODMAS — total ?; splits: 30/30 (32k train, 40k test) (-1), 50/30 (54k train, 67k test) (-1), 70/30 (75k train, 94k test) (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified instances out of the total number of instances in the test set.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the model's ability to rank positive instances higher than negative ones across all classification thresholds.

Input / output format

Input: VAE-derived latent space feature vectors representing malware samples (dimensionally reduced, invariant features extracted from raw data).

Output: Predicted malware class label (multi-class classification).

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    accuracy = (y_true == y_pred).mean()
    auc = roc_auc_score(y_true, y_prob, multi_class='ovr') if y_prob is not None else None
    return {'accuracy': accuracy, 'auc': auc}

Common pitfalls

• Performance is highly sensitive to the training-test split ratio, particularly for Decision Trees and Naive Bayes, so results cannot be directly compared across different partition schemes.
• Random Forest shows significant sensitivity to random seed initialization in specific splits (e.g., EMBER 50/30), unlike other classifiers which remain stable across seeds 42 and 123.
• The evaluation explicitly excludes hyperparameter tuning; comparing against tuned baselines or reporting tuned results violates the stated protocol.

Evidence (verbatim from paper)

The experiments rigorously evaluate model accuracy across different training-test split ratios (30/30, 50/30, and 70/30) and random seeds (42 and 123) to assess robustness. A key novelty of this approach is that no hyperparameter tuning was required, yet the models achieved high performance directly from latent space features... Random Forest exhibited the best classification performance overall. With a 30/30 split, it achieved cross-validation scores of 0.9498, test accuracy of 0.9523, and an AUC close to 0.9906.

Citation

@misc{ajayi2025vae,
  title={Leveraging VAE-Derived Latent Spaces for Enhanced Malware Detection with Machine Learning Classifiers},
  author={Ajayi et al. (2025)},
  year={2025},
  note={arXiv:2503.20803}
}

• arXiv: 2503.20803