Continual Learning Malware Eval

continual-learning-malware-eval

On the Limitations of Continual Learning for Malware Classification — Rahman et al. (2022) (arXiv:2208.06568, 2022)

What this evaluates

Evaluates continual learning techniques for malware classification under domain, class, and task incremental settings. It measures how well models adapt to evolving malware distributions without catastrophic forgetting. The protocol compares complex CL methods against simple baselines like joint replay.

Datasets

• Drebin — total ?; splits: train (-1)
• EMBER — total ?; splits: train (-1)

Metrics

• Mean accuracy (primary) — range: percent
  • Average of the accuracy scores across all tasks in a given experiment.
• Min accuracy — range: percent
  • The minimum accuracy score among all individual tasks in a given experiment, used to highlight catastrophic forgetting on the weakest task.

Input / output format

Input: Malware binary samples (feature extraction not explicitly detailed in the provided text; standard malware classification input).

Output: Classification label (e.g., malware vs. goodware, or specific malware family).

Scoring recipe

accuracies = []
for task in tasks:
    correct = sum(1 for pred, gold in zip(predictions[task], golds[task]) if pred == gold)
    accuracies.append(correct / len(golds[task]))
mean_acc = sum(accuracies) / len(accuracies)
min_acc = min(accuracies)

Common pitfalls

• Focusing solely on Mean accuracy obscures catastrophic forgetting; Min accuracy is required to identify the weakest-performing task.
• Assuming complex continual learning methods outperform simple joint replay, which the protocol shows is often false for high-volume, non-stationary malware data.

Evidence (verbatim from paper)

The results of each of the experiments are represented in both Mean and Min metrics. Mean represents the mean accuracy of all the tasks in a single experiment, such as across 10, 12, 14, 16, and 18 classes tested for Class-IL with a given CL method on Drebin. Similarly, Min represents the minimum accuracy among all the tasks, which we highlight because the weakest performance shows the degree to which a technique may not be suitable for use.

Citation

@misc{rahman2022limitations,
  title={On the Limitations of Continual Learning for Malware Classification},
  author={Rahman et al. (2022)},
  year={2022},
  note={arXiv:2208.06568}
}

• arXiv: 2208.06568