Ember2024 Eval

ember2024-eval

EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers — Joyce et al. (2025) (arXiv:2506.05074, 2025)

What this evaluates

Evaluates malware classifiers on detection, family identification, and attribute prediction tasks across multiple file formats. It specifically probes robustness against concept drift and novel malware families using a temporally separated test set and a challenge set of evasive samples.

Datasets

• EMBER2024 — total 3200000; splits: train (-1), test (-1), challenge (-1); repo https://github.com/FutureComputing4AI/EMBER2024

Metrics

• ROC AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• PR AUC — range: [0, 1]
  • Area under the Precision-Recall curve, measuring the trade-off between precision and recall across thresholds. Preferred over ROC AUC for highly imbalanced datasets.

Input / output format

Input: Malware or benign files in PE (Win32/Win64/.NET), APK, ELF, or PDF formats, represented using EMBER feature version 3.

Output: Binary label (malicious/benign) or multi-class/multi-label predictions for family/attributes.

Scoring recipe

def compute_roc_auc(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_pr_auc(y_true, y_scores):
    precision, recall, _ = precision_recall_curve(y_true, y_scores)
    return auc(recall, precision)

Common pitfalls

• ROC AUC can be misleading on highly imbalanced datasets (e.g., the challenge set); PR AUC should be reported instead.
• The challenge set contains only malware, so it must be joined with the corresponding benign test partition to compute standard binary classification metrics.
• Temporal train/test splits introduce concept drift, making detection of novel families significantly harder than random splits.

Evidence (verbatim from paper)

The ROC AUC and Precision-Recall (PR) AUC scores in Table 5 indicate that the trained LightGBM classifiers are able to accurately detect malicious files in the EMBER2024 test set, which consists of files that appeared in VirusTotal for the first time 1-12 weeks after the most recent file in the training set.

Citation

@misc{joyce2025ember2024,
  title={EMBER2024 -- A Benchmark Dataset for Holistic Evaluation of Malware Classifiers},
  author={Joyce et al. (2025)},
  year={2025},
  note={arXiv:2506.05074}
}

• arXiv: 2506.05074