Microsoft Malware Classification Eval

microsoft-malware-classification-eval

Fusing Feature Engineering and Deep Learning: A Case Study for Malware Classification — Gibert et al. (2022) (arXiv:2206.05735, 2022)

What this evaluates

Evaluates the ability of hybrid machine learning models to classify Windows malware into specific family categories by fusing hand-crafted structural features with deep learning-derived representations. It probes robustness against class imbalance and measures both categorical correctness and probabilistic calibration.

Datasets

• Microsoft Malware Classification Challenge Dataset — total 21741; splits: train (10868), test (10873)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correct predictions: Number of correct predictions / Total number of predictions.
• logloss — range: [0, ∞)
  • Multi-class logarithmic loss (cross-entropy): -1/N ∑{i=1}^N ∑{j=1}^M y_{i,j} log(p_{i,j}), where y is one-hot true label and p is predicted probability.

Input / output format

Input: Hexadecimal representation of malware binary content and corresponding assembly language source code, processed into concatenated hand-crafted and deep feature vectors.

Output: Predicted malware family label (one of 9 classes) and a probability distribution over all 9 classes.

Scoring recipe

def compute_metrics(y_true, y_pred_proba, y_pred_label):
    # y_true: one-hot encoded labels (N, 9)
    # y_pred_proba: predicted probabilities (N, 9)
    # y_pred_label: predicted class indices (N,)
    accuracy = np.mean(y_true.argmax(axis=1) == y_pred_label)
    N = y_true.shape[0]
    logloss = -np.sum(y_true * np.log(y_pred_proba + 1e-15)) / N
    return {'accuracy': accuracy, 'logloss': logloss}

Common pitfalls

• The dataset is highly imbalanced across the 9 malware families, making accuracy alone an unreliable indicator of model robustness.
• Test set ground-truth labels are withheld (Kaggle competition format), requiring submission of predicted probabilities to compute logloss rather than direct label comparison.
• Feature extraction relies on proprietary tools (IDA Pro for disassembly) and custom CNN pipelines, which are not fully open-sourced, complicating exact replication.

Evidence (verbatim from paper)

Regarding the performance metrics used to evaluate our approach, we will report two metrics, the accuracy and the logarithmic loss. The accuracy is simply the fraction of correct predictions. Formally, accuracy is defined as follows: accuracy = Number of correct predictions / Total number of predictions. However, accuracy alone is not a good evaluation metric to assess the robustness of machine learning models in datasets where there exist a large class imbalance. Subsequently, the multi-class logarithmic loss (logloss) has been used to to assess the performance of the predictions. The logarithmic loss is the cross entropy between the distribution of true labels and the predicted probabilities. Formally, it is defined as follows: logloss = -1/N ∑{i=1}^{N}∑{j=1}^{M}y_{i,j} log(p_{i,j})

Citation

@misc{gibert2022fusing,
  title={Fusing Feature Engineering and Deep Learning: A Case Study for Malware Classification},
  author={Gibert et al. (2022)},
  year={2022},
  note={arXiv:2206.05735}
}

• arXiv: 2206.05735