Melemad Eval

melemad-eval

MeLeMaD: Adaptive Malware Detection via Chunk-wise Feature Selection and Meta-Learning — Ajvad Haneef K et al. (2025) (arXiv:2512.23987, 2025)

What this evaluates

This evaluation protocol assesses a model's ability to classify Android and Windows PE binaries as benign or malicious using only static features. It probes robustness, generalization across malware families, and discriminative power under concept drift and evolving threat scenarios.

Datasets

• CIC-AndMal2020 — total 400000; splits: train (-1), test (-1)
• BODMAS — total 134435; splits: train (-1), test (-1)
• EMBOD — total 934311; splits: train (-1), test (-1); repo https://www.kaggle.com/datasets/ajvadhaneef/embod-all/

Metrics

• Accuracy (primary) — range: [0, 1]
  • Proportion of correct predictions out of total predictions: (TP+TN)/(TP+TN+FP+FN).
• Precision — range: [0, 1]
  • Proportion of true positive predictions among all positive predictions: TP/(TP+FP).
• Recall — range: [0, 1]
  • Proportion of true positive predictions out of all actual positive instances: TP/(TP+FN).
• F1-Score — range: [0, 1]
  • Harmonic mean of precision and recall: 2*(Precision*Recall)/(Precision+Recall).
• MCC — range: [0, 1]
  • Balanced metric for binary classification considering all confusion matrix quadrants: (TPTN - FPFN)/sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)).
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, integrating True Positive Rate over False Positive Rate across thresholds.

Input / output format

Input: Pre-extracted static feature vectors representing Android applications (permissions, API calls, intents, strings) or Windows PE binaries (headers, imports, sections, entropy statistics), formatted as numerical arrays. Labels are binary (0 for benign, 1 for malware).

Output: Binary classification label (0 or 1) or continuous probability score for threshold-dependent metrics and AUC-ROC calculation.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    tp = np.sum((y_true == 1) & (y_pred == 1))
    tn = np.sum((y_true == 0) & (y_pred == 0))
    fp = np.sum((y_true == 0) & (y_pred == 1))
    fn = np.sum((y_true == 1) & (y_pred == 0))
    acc = (tp + tn) / (tp + tn + fp + fn)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    mcc = ((tp*tn) - (fp*fn)) / np.sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))
    auc = roc_auc_score(y_true, y_prob) if y_prob is not None else None
    return acc, prec, rec, f1, mcc, auc

Common pitfalls

• The paper explicitly restricts training to static features only, ignoring dynamic features (system calls, network traffic) present in the raw datasets.
• Datasets are class-balanced (50/50 benign/malware), but this does not represent the train/validation/test split ratios used during meta-learning experiments.
• AUC-ROC requires predicted probabilities, not hard class labels, yet the paper reports it alongside accuracy without clarifying the thresholding strategy.

Evidence (verbatim from paper)

We evaluate our proposed method in terms of Accuracy($A_{c}$), Precision($P_{c}$), Recall($R_{c}$), F1 score($F1_{c}$), Matthews Correlation Coefficient($M_{c}$), and Area Under the Receiver Operating Characteristic Curve ($AUC-ROC$). These metrics are calculated based on the number of True Positives (TP), True Negatives (TN), False Positives (FP), and False Negatives (FN).

Citation

@misc{haneef2025melemad,
  title={MeLeMaD: Adaptive Malware Detection via Chunk-wise Feature Selection and Meta-Learning},
  author={Ajvad Haneef K et al. (2025)},
  year={2025},
  note={arXiv:2512.23987}
}

• arXiv: 2512.23987