Ember Eval

ember-eval

EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models — Anderson et al. (2018) (arXiv:1804.04637, 2018)

What this evaluates

Evaluates machine learning models on static malware classification of Windows PE binaries. It probes the effectiveness of engineered static features versus raw binary inputs for distinguishing malicious from benign software.

Datasets

• EMBER — total ?; splits: train (-1), test (-1); repo https://github.com/endgameinc/ember

Metrics

• ROC AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the probability that a randomly chosen malicious sample scores higher than a randomly chosen benign sample across all classification thresholds.
• detection rate — range: percent
  • True positive rate (recall) at a specified false positive rate threshold, commonly reported at 0.1% and 1% FPR.

Input / output format

Input: Vectorized static features extracted from Windows PE binaries, formatted as a 2351-dimensional vector using raw features and feature hashing.

Output: Binary classification label: malicious or benign.

Scoring recipe

def evaluate(y_true, y_scores):
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    roc_auc = auc(fpr, tpr)
    idx_01 = np.argmin(np.abs(fpr - 0.001))
    det_rate_01 = tpr[idx_01]
    idx_1 = np.argmin(np.abs(fpr - 0.01))
    det_rate_1 = tpr[idx_1]
    return roc_auc, det_rate_01, det_rate_1

Common pitfalls

• Using the J48/Adobe Malware Classifier as a baseline without recognizing its dataset bias toward OS classification rather than malware detection.
• Expecting featureless deep learning models (e.g., MalConv) to outperform traditional ML on structured PE features without significant hyperparameter tuning.
• Reporting detection rates without specifying the exact false positive rate threshold, as performance varies significantly between 0.1% and 1% FPR.

Evidence (verbatim from paper)

The ROC AUC exceeds 0.99911. A threshold of 0.871 on the model score results in less than 0.1% FP rate at a detection rate exceeding 92.99%. At less than 1% FP rate, the model exceeds 98.2% detection rate.

Citation

@misc{anderson2018ember,
  title={EMBER: An Open Dataset for Training Static PE Malware Machine Learning Models},
  author={Anderson et al. (2018)},
  year={2018},
  note={arXiv:1804.04637}
}

• arXiv: 1804.04637