Malware Behavioral Report Eval

malware-behavioral-report-eval

Behavioural Reports of Multi-Stage Malware — Carpenter et al. (2023) (arXiv:2301.12800, 2023)

What this evaluates

Evaluates host-based intrusion detection models on their ability to classify multi-label malware behaviors from truncated Windows API call sequences. Probes how well different neural architectures handle sequential behavioral data and feature selection strategies for detecting overlapping malicious activities.

Datasets

• Behavioural Reports of Multi-Stage Malware — total ?; splits: train (-1), test (-1); repo https://github.com/marcusCarpenter97/Malware-data

Metrics

• F1-score (primary) — range: [0, 1]
  • F1 = 2 × (precision × recall) ÷ (precision + recall). Computed with average='samples' for a global score or average=None for per-class scores, using zero_division=1.
• Binary accuracy — range: [0, 1]
  • Element-wise accuracy calculated by Tensorflow, matching each digit in the binary label array separately rather than requiring exact set matches.
• Precision — range: [0, 1]
  • Standard precision metric computed with average='samples' or per-class, measuring the proportion of correctly predicted positive labels.
• Recall — range: [0, 1]
  • Standard recall metric computed with average='samples' or per-class, measuring the proportion of actual positive labels correctly identified.
• Binary cross entropy — range: other
  • Log loss function used for model optimization, measuring the difference between predicted probabilities and binary labels.

Input / output format

Input: Truncated sequences of 512 Windows API calls (first 512, last 512, or random 512 consecutive calls from the full sequence), processed as feature vectors for sequential neural networks.

Output: Multi-label binary vector indicating the presence or absence of specific malware behaviors (e.g., Benign, Dropper, Spyware, Miner, Hacktool, Fakeav, Pua, Downloader, Adware, Ransomware, Worm, Virus, Trojan, Banker, Generic).

Scoring recipe

import numpy as np
from sklearn.metrics import f1_score, precision_score, recall_score

# predictions: (N, C) binary array, labels: (N, C) binary array
exact_acc = np.mean(np.all(predictions == labels, axis=1))
bin_acc = np.mean(predictions == labels)
f1_samples = f1_score(labels, predictions, average='samples', zero_division=1)
f1_per_class = f1_score(labels, predictions, average=None, zero_division=1)
prec = precision_score(labels, predictions, average='samples', zero_division=1)
rec = recall_score(labels, predictions, average='samples', zero_division=1)
bce = -np.mean(labels * np.log(predictions + 1e-7) + (1 - labels) * np.log(1 - predictions + 1e-7))

Common pitfalls

• Confusing Scikit-learn's exact-match accuracy (requires all labels to match) with Tensorflow's element-wise binary accuracy, which is heavily inflated by label sparsity.
• Using standard macro/micro averaging for F1 instead of the paper's specified average='samples' or average=None with zero_division=1.
• Ignoring the three data slicing strategies (First, Last, Random 512 APIs) which drastically change model performance and must be reported separately.

Evidence (verbatim from paper)

The accuracy, F1-score, precision, recall, and the binary cross entropy which was used as the loss function for optimising the models. There are only five metrics in the list because Scikit learn and Tensorflow provide different accuracy measurements. Scikit learn computes the F1-score as: F1=2×(precision×recall)÷(precision+recall)

Citation

@misc{carpenter2023behavioural,
  title={Behavioural Reports of Multi-Stage Malware},
  author={Carpenter et al. (2023)},
  year={2023},
  note={arXiv:2301.12800}
}

• arXiv: 2301.12800