Iot Malware Image Classification Eval

iot-malware-image-classification-eval

Lightweight Classification of IoT Malware based on Image Recognition — Su et al. (2018) (arXiv:1802.03714, 2018)

What this evaluates

Evaluates the ability of a lightweight CNN to classify IoT binary files as benign or belonging to specific DDoS malware families (Mirai, Linux.Gafgyt) by converting raw binaries into 64x64 grayscale images.

Datasets

• IoTPOT IoT DDoS Malware Dataset — total 365; splits: train (320), test (45); repo https://github.com/Carina02/IotMalwareImage

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified samples out of the total test samples. Reported for both 2-class (benign vs. malicious) and 3-class (benign, Mirai, Gafgyt) settings.

Input / output format

Input: Raw binary file converted to a 64x64 one-channel grayscale image, where each pixel value corresponds to the decimal encoding of an 8-bit byte from the binary.

Output: Class label: 'Benign', 'Malicious' (2-class), or 'Benign', 'Gafgyt', 'Mirai' (3-class).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return (correct / len(gold)) * 100
# Average the accuracy over 5 independent random train/test splits

Common pitfalls

• The dataset was manually balanced to exactly 365 samples (15 test per class); using the original unbalanced IoTPOT collection will change results.
• Evaluation is averaged over 5 independent random splits, not a standard fixed k-fold or single hold-out set.
• Input must be strictly 64x64 grayscale; resizing or using RGB channels breaks the protocol.

Evidence (verbatim from paper)

The experiments were conducted five times which each time with a completely different training/test data combination (i.e., there are no shared test samples between any two of five test data sets). According to the results of two-class classification, we find the proposed system can predict the existence of maliciousness with about 94.0% accuracy on the average.

Citation

@misc{su2018lightweight,
  title={Lightweight Classification of IoT Malware based on Image Recognition},
  author={Su et al. (2018)},
  year={2018},
  note={arXiv:1802.03714}
}

• arXiv: 1802.03714