Opcode Malware Classification Eval

opcode-malware-classification-eval

OpCode-Based Malware Classification Using Machine Learning and Deep Learning Techniques — Saini et al. (2025) (arXiv:2504.13408, 2025)

What this evaluates

This evaluation probes the ability of machine learning and deep learning models to classify malware into specific families based on their assembly-level instruction sequences (opcodes). It compares traditional feature-engineering approaches using 1-gram and 2-gram n-grams against an end-to-end 1D CNN that processes raw opcode sequences.

Datasets

• OpCode Malware Dataset — total ?; splits: test (-1)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of correctly classified instances out of the total number of instances.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall).

Input / output format

Input: Traditional models: normalized 1-gram and 2-gram n-gram feature vectors derived from opcode sequences. CNN: raw opcode sequences.

Output: Predicted malware family label (categorical class).

Scoring recipe

def compute_metrics(predictions, gold):
    accuracy = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold)
    classes = set(gold)
    f1_scores = []
    for c in classes:
        tp = sum(1 for p, g in zip(predictions, gold) if p == c and g == c)
        fp = sum(1 for p, g in zip(predictions, gold) if p == c and g != c)
        fn = sum(1 for p, g in zip(predictions, gold) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
        f1_scores.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0)
    f1 = sum(f1_scores) / len(f1_scores)
    return {'accuracy': accuracy, 'f1_score': f1}

Common pitfalls

• Class imbalance was explicitly mitigated using RandomOverSampler during training, so evaluation metrics should ideally be reported on the original imbalanced test set to reflect real-world deployment conditions.
• The protocol compares handcrafted n-gram features against raw sequence inputs, which can confound whether performance differences stem from the model architecture or the feature representation strategy.

Evidence (verbatim from paper)

SVM: Accuracy 66.37%, F1-score 64.04%

• KNN: Accuracy 63.65%, F1-score 61.02%
  Decision Tree: Accuracy 62.14%, F1-score 60.06%
• Voting Classifier: Accuracy: $68.61%$ (but did not outperform SVM).

Deep Learning Approach:

The CNN is trained using PyTorch for 10 epochs with the Adam optimizer $(\mathrm{lr} = 0.001)$ and a ReduceLROnPlateau scheduler. Evaluation metrics for the CNN are:

Accuracy: $62.14%$

• Precision: ${64.49}%$
• Recall: $62.14%$
  F1-score: $60.44%$

Comparative analysis indicates that while the CNN reduces the need for manual feature engineering, SVM remains the top-performing model on this dataset.

Citation

@misc{saini2025opcode,
  title={OpCode-Based Malware Classification Using Machine Learning and Deep Learning Techniques},
  author={Saini et al. (2025)},
  year={2025},
  note={arXiv:2504.13408}
}

• arXiv: 2504.13408