Ember Malware Detection Eval

ember-malware-detection-eval

Evaluating Ensemble and Deep Learning Models for Static Malware Detection with Dimensionality Reduction Using the EMBER Dataset — Abedin et al. (2025) (arXiv:2507.16952, 2025)

What this evaluates

Evaluates the performance and robustness of various machine learning classifiers for static malware detection on high-dimensional tabular features, specifically assessing the impact of dimensionality reduction techniques (PCA, LDA) on classification accuracy and discriminative power.

Datasets

• EMBER — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly classified instances out of the total number of instances.
• precision — range: [0, 1]
  • Ratio of true positive predictions to the total number of positive predictions.
• recall — range: [0, 1]
  • Ratio of true positive predictions to the total number of actual positive instances.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall.
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes across all classification thresholds.

Input / output format

Input: High-dimensional tabular feature vectors extracted from PE files (static malware features).

Output: Binary classification label (malware/benign) or probability scores for each class.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    accuracy = (y_true == y_pred).mean()
    precision = precision_score(y_true, y_pred)
    recall = recall_score(y_true, y_pred)
    f1 = f1_score(y_true, y_pred)
    auc = roc_auc_score(y_true, y_prob) if y_prob is not None else None
    return {'accuracy': accuracy, 'precision': precision, 'recall': recall, 'f1': f1, 'auc': auc}

Common pitfalls

• KNN performance heavily degrades in high-dimensional spaces due to distance distortion, requiring dimensionality reduction.
• LDA significantly harms tree-based ensemble models by compressing non-linear decision boundaries into a single axis.
• TabNet is highly sensitive to feature compression and may underperform under PCA without extensive tuning.
• The evaluation only covers static features and does not test adversarial evasion or concept drift over time.

Evidence (verbatim from paper)

We evaluate classification accuracy, precision, recall, F1-score, and AUC to understand the models' generalizability, robustness, and discriminative capacity.

Citation

@misc{abedin2025evaluating,
  title={Evaluating Ensemble and Deep Learning Models for Static Malware Detection with Dimensionality Reduction Using the EMBER Dataset},
  author={Abedin et al. (2025)},
  year={2025},
  note={arXiv:2507.16952}
}

• arXiv: 2507.16952