Deepmal Malware Detection Eval

deepmal-malware-detection-eval

DeepMAL -- Deep Learning Models for Malware Traffic Detection and Classification — Marín et al. (2020) (arXiv:2003.04079, 2020)

What this evaluates

Evaluates deep learning models for binary malware traffic detection using raw network bytestreams. It probes the model's ability to distinguish benign from malicious network flows or packets without relying on handcrafted domain features.

Datasets

• USTCTFC2016 — total 1000000; splits: train (800000), val (100000), test (100000)

Metrics

• accuracy (primary) — range: percent
  • Proportion of correctly classified instances (benign and malware) out of the total test set. Calculated as (TP + TN) / Total.
• AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• false_alarm_rate — range: percent
  • Ratio of false positives to all actual negative (benign) instances. Calculated as FP / (FP + TN).

Input / output format

Input: Raw network packet or flow byte streams. Packets are trimmed/padded to 1024 bytes. Flows use the first 2 packets, each trimmed to 100 bytes. Bytes are represented as decimal normalized values.

Output: Binary classification label: 'benign' or 'malware'.

Scoring recipe

def compute_metrics(predictions, labels):
    tp = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 1)
    tn = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 0)
    fp = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 0)
    fn = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 1)
    total = tp + tn + fp + fn
    accuracy = (tp + tn) / total
    far = fp / (fp + tn) if (fp + tn) > 0 else 0.0
    return {'accuracy': accuracy, 'false_alarm_rate': far}

Common pitfalls

• The dataset is artificially balanced (50% benign, 50% malware), which inflates accuracy compared to real-world imbalanced traffic.
• Packet-level raw inputs perform poorly; flow-level representations are required for high accuracy and practical deployment.
• Comparing against shallow models requires using identical raw inputs, not handcrafted features, to ensure a fair evaluation of representation learning.

Evidence (verbatim from paper)

The performance metric chosen in both cases was the accuracy, since the dataset is balanced. For the Raw Packets representation we achieved 77.6% of accuracy over the test set; while in the case of Raw Flows we achieved an accuracy of 98.6% also over the test set.

Citation

@misc{marin2020deepmal,
  title={DeepMAL -- Deep Learning Models for Malware Traffic Detection and Classification},
  author={Marín et al. (2020)},
  year={2020},
  note={arXiv:2003.04079}
}

• arXiv: 2003.04079