Mad Ood Eval

mad-ood-eval

MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification — Ige et al. (2025) (arXiv:2512.17594, 2025)

What this evaluates

Evaluates a model's ability to detect out-of-distribution (OOD) malware variants and classify known malware families without using OOD samples during training. It probes both classification accuracy on in-distribution data and the statistical separation capability between known and novel threats using cluster-driven decision boundaries.

Datasets

• Unspecified malware dataset (25 families) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUROC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the probability that a randomly chosen in-distribution sample is ranked higher than a randomly chosen OOD sample by the model's confidence or distance score.
• Accuracy — range: [0, 1]
  • Standard classification accuracy: the ratio of correctly predicted in-distribution samples (benign or malware family) to the total number of in-distribution test samples.

Input / output format

Input: Image representations of malware samples. For the second-stage classifier, inputs also include the initial cluster analysis prediction and the first model's prediction output.

Output: Multi-class classification label (benign or specific malware family) and an OOD detection score/probability derived from Z-score distances to class centroids.

Scoring recipe

def compute_auroc(y_true, y_scores):
    # y_true: 1 for in-distribution, 0 for OOD
    # y_scores: model confidence or inverse distance for in-distribution class
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

def compute_accuracy(y_true, y_pred):
    return sum(y_true == y_pred) / len(y_true)

Common pitfalls

• The Z-score thresholding (±1) is an internal decision rule for flagging outliers during inference, not the evaluation metric itself; evaluation relies on AUROC.
• OOD samples are explicitly excluded from training; using them during training violates the paper's protocol of learning only from in-distribution data.
• The test set contains both in-distribution and OOD samples, so AUROC must be computed on the combined test set, not just on in-distribution data.

Evidence (verbatim from paper)

Evaluation Metrics In-Distribution Performance: Accuracy, and Confusion matrix for malware classification. OOD Detection Performance: AUROC (Area Under ROC Curve): Measures separation between in-distribution and OOD

Citation

@misc{ige2025madood,
  title={MAD-OOD: A Deep Learning Cluster-Driven Framework for an Out-of-Distribution Malware Detection and Classification},
  author={Ige et al. (2025)},
  year={2025},
  note={arXiv:2512.17594}
}

• arXiv: 2512.17594