Malware Image Classification Eval

malware-image-classification-eval

Deep Multi-Task Learning for Malware Image Classification — Bensaoud et al. (2024) (arXiv:2405.05906, 2024)

What this evaluates

Evaluates a deep learning model's ability to classify malware images across multiple tasks, including binary classification, malware family classification, and detection of obfuscation techniques across Windows, Android, macOS, and Linux platforms.

Datasets

• Maling benchmark dataset — total ?; splits: test (-1); repo https://github.com/abensaou-uccs/Colorado-MalColorImg

Metrics

• accuracy (primary) — range: percent
  • Calculated as the percentage of correctly classified samples out of the total test samples. Accuracy = (Correct Predictions / Total Samples) * 100.

Input / output format

Input: Bitmap (BMP/PNG) images representing structural features of PE, APK, Mach-O, and ELF binaries.

Output: Discrete class labels corresponding to one of seven classification tasks (binary classification, malware family classification, or obfuscation detection).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100

Common pitfalls

• The paper does not specify the exact train/validation/test split ratios or data augmentation pipeline details beyond CycleGAN for macOS, making exact reproduction of error rates difficult.
• Tasks 3 (iOS) and 6 (macOS) explicitly lack state-of-the-art comparisons due to a scarcity of prior image-based research, which limits the completeness of the benchmark evaluation.

Evidence (verbatim from paper)

For the malware image binary classification task1, we obtain an accuracy of 99.88%, 99.94% for task2, 99.91% for task3, 99.89% for task5, 99.92% for task6, and 99.95% for task7, while task4 for the malware family classification yields a classification accuracy of 99.97% as shown in Table 5.

Citation

@misc{bensaoud2024deep,
  title={Deep Multi-Task Learning for Malware Image Classification},
  author={Bensaoud et al. (2024)},
  year={2024},
  note={arXiv:2405.05906}
}

• arXiv: 2405.05906