Malware Ransomware Detection Eval

malware-ransomware-detection-eval

AI-based Malware and Ransomware Detection Models — Marais et al. (2022) (arXiv:2207.02108, 2022)

What this evaluates

Evaluates the ability of machine learning and deep learning models to classify Windows PE binaries as benign or malicious, and further categorize malicious samples into specific families or ransomware types using static analysis features and grayscale image representations.

Datasets

• Ember — total 800000; splits: unspecified (-1)
• Bodmas — total 134435; splits: unspecified (-1)
• PEMachineLearning — total 201549; splits: unspecified (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Standard classification accuracy (correct predictions divided by total predictions). Not explicitly detailed in the provided section.

Input / output format

Input: Static analysis features (2,381-dimensional vector via Ember extractor) OR 64×64 grayscale images (via Grayscale extractor) derived from Windows PE binary files.

Output: Binary classification label (benign vs. malicious) followed by multi-class family/ransomware label for malicious samples.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

# For multi-stage evaluation:
# stage1_acc = compute_accuracy(benign_mal_pred, true_benign_mal)
# stage2_acc = compute_accuracy(family_pred, true_family)  # only on malicious samples

Common pitfalls

• Dataset formats vary significantly: Ember and Bodmas provide pre-extracted feature vectors, while PEMachineLearning and Bodmas also provide raw PE files requiring different preprocessing pipelines.
• Class imbalance is severe in Bodmas (e.g., only 821 ransomware samples vs. 57,293 total malicious files), which can skew family-level evaluation and require stratified sampling or weighted loss.
• Grayscale image conversion (64×64) discards fine-grained binary structure, potentially causing CNNs to underperform compared to feature-based models like LightGBM/XGBoost.

Evidence (verbatim from paper)

For our experiments, we rely on three different datasets. They all contain malicious and benign Portable Executable (PE) files in two different formats. ... To train our models, we choose to resize grayscale images 64×64 pixels.

Citation

@misc{marais2022aimalware,
  title={AI-based Malware and Ransomware Detection Models},
  author={Marais et al. (2022)},
  year={2022},
  note={arXiv:2207.02108}
}

• arXiv: 2207.02108