qhjqhj00 malimg-classification-eval
CNN vs ELM for Image-Based Malware Classification — Jain et al. (2021) (arXiv:2103.13820, 2021)
What this evaluates
This benchmark evaluates the ability of machine learning models to classify malware binaries by converting them into grayscale images and predicting their specific family among 25 categories. It probes classification accuracy and computational efficiency across different neural network architectures.
Datasets
- Malimg — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: [0, 1]- The proportion of correctly classified samples out of the total number of samples. Calculated as the number of correct predictions divided by the total number of predictions.
Input / output format
Input: Grayscale images of malware binaries generated via byte-to-pixel mapping (sizes: 32×32, 64×64, or 128×128 pixels), or 1D feature vectors of fixed length (512 or 1024) for ELM experiments.
Output: A single class label corresponding to one of the 25 malware families in the Malimg dataset.
Scoring recipe
def compute_accuracy(predictions, gold_labels):
correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
return correct / len(gold_labels)
Common pitfalls
- The Malimg dataset is highly imbalanced across its 25 classes, which can bias standard accuracy metrics toward majority classes.
- ELM results are averaged over 50 independent runs due to random weight initialization, so single-run evaluations will show high variance.
- Input image resolution significantly affects CNN accuracy but has minimal impact on 1D vector-based ELM performance.
Evidence (verbatim from paper)
For each combination of parameters, we train and test 50 ELMs, and evaluate the performance in terms of the average accuracy.
Citation
@misc{jain2021cnneelm,
title={CNN vs ELM for Image-Based Malware Classification},
author={Jain et al. (2021)},
year={2021},
note={arXiv:2103.13820}
}
- arXiv: 2103.13820