Big2015 Malware Detection Eval

big2015-malware-detection-eval

Deep Learning Fusion For Effective Malware Detection: Leveraging Visual Features — Johny et al. (2024) (arXiv:2405.14311, 2024)

What this evaluates

Evaluates the effectiveness of multimodal visual feature fusion (grayscale, entropy graph, SimHash) using VGG16 for binary malware classification and family detection. It probes the model's ability to handle imbalanced malware datasets and detect obfuscated binaries.

Datasets

• BIG2015 — total ?; splits: test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • F1 = 2 * (Precision * Recall) / (Precision + Recall), where Precision = TP / (TP + FP) and Recall = TP / (TP + FN).
• Accuracy — range: [0, 1]
  • Accuracy = (TP + TN) / (TP + FP + TN + FN).
• Precision — range: [0, 1]
  • Precision = TP / (TP + FP).
• Recall — range: [0, 1]
  • Recall = TP / (TP + FN).
• Prediction Time — range: seconds
  • Wall-clock time in seconds required to predict a single file.

Input / output format

Input: Hexadecimal binary content of malware files (header stripped), converted into grayscale images, entropy graphs, and SimHash images.

Output: Predicted malware family/class label and inference time in seconds per file.

Scoring recipe

def compute_metrics(tp, tn, fp, fn):
    accuracy = (tp + tn) / (tp + fp + tn + fn)
    precision = tp / (tp + fp)
    recall = tp / (tp + fn)
    f1 = 2 * (precision * recall) / (precision + recall)
    return accuracy, precision, recall, f1

Common pitfalls

• Dataset is imbalanced, so Accuracy can be misleading; F1-score is explicitly prioritized.
• Visual features are generated via a custom hex-to-image conversion pipeline not fully detailed, which may hinder exact reproducibility.
• Experiments are repeated 5 times and averaged to minimize random error, but variance is not reported.

Evidence (verbatim from paper)

Given the imbalanced nature of the dataset, we prioritize the F1-score over other metrics to assess the model's performance. Besides the above-mentioned metrics, we also estimate the time required to predict a file using our proposed models.

Citation

@misc{johny2024deeplearningfusion,
  title={Deep Learning Fusion For Effective Malware Detection: Leveraging Visual Features},
  author={Johny et al. (2024)},
  year={2024},
  note={arXiv:2405.14311}
}

• arXiv: 2405.14311