Malicia Malware Classification Eval

malicia-malware-classification-eval

A Natural Language Processing Approach to Malware Classification — Mehta et al. (2023) (arXiv:2307.11032, 2023)

What this evaluates

Evaluates a model's ability to classify malware samples into one of seven predefined families based on their opcode sequences. It probes the model's capacity to capture structural and sequential patterns in low-level code representations.

Datasets

• Malicia — total 8054; splits: validation (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified samples out of the total number of samples.
• weighted F1-score — range: [0, 1]
  • Weighted average of per-class F1 scores, where weights correspond to the number of true instances for each class.

Input / output format

Input: Opcode sequences extracted from malware samples, truncated to a fixed length L (25, 50, 100, or 200). Samples with fewer opcodes than L are dropped.

Output: A single class label from the seven malware families.

Scoring recipe

def compute_metrics(predictions, gold_labels):
    accuracy = sum(p == g for p, g in zip(predictions, gold_labels)) / len(gold_labels)
    classes = set(gold_labels)
    f1_scores, weights = [], []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold_labels) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold_labels) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
        weights.append(sum(1 for g in gold_labels if g == c))
    weighted_f1 = sum(f * w for f, w in zip(f1_scores, weights)) / sum(weights)
    return accuracy, weighted_f1

Common pitfalls

• The dataset is highly imbalanced across the seven malware families, so overall accuracy can be misleading for minority classes.
• Samples with opcode sequences shorter than the chosen truncation length L are dropped, which may slightly alter the effective dataset size and distribution.
• The paper reports validation accuracy for hyperparameter tuning rather than a held-out test set, so results may reflect tuning bias.

Evidence (verbatim from paper)

The accuracy we obtained for the best choice of hyperparameters in Table 2 was 0.9758. Table 4 shows the accuracy and weighted F1-score obtained after testing the following techniques on the same seven families of the Malicia dataset.

Citation

@misc{mehta2023malware,
  title={A Natural Language Processing Approach to Malware Classification},
  author={Mehta et al. (2023)},
  year={2023},
  note={arXiv:2307.11032}
}

• arXiv: 2307.11032