Propedeutica Malware Detection Eval

propedeutica-malware-detection-eval

Learning Fast and Slow: PROPEDEUTICA for Real-time Malware Detection — Sun et al. (2017) (arXiv:1712.01145, 2017)

What this evaluates

Evaluates a two-stage malware detection framework that uses a fast ML classifier for initial triage and a deep learning model for borderline cases. It probes the model's ability to accurately classify system call sequences as malicious or benign while balancing detection latency and false positive rates in real-time scenarios.

Datasets

• Propedeutica System Call Dataset — total 493095; splits: train (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Number of correctly classified samples divided by the total number of samples.
• F1 Score — range: [0, 1]
  • Harmonic mean of precision and recall.
• false positive rate — range: [0, 1]
  • Number of false positives divided by the total number of actual negatives.

Input / output format

Input: Sliding windows of 100 system calls (stride 50) extracted from 5-minute execution traces of Windows PE32 binaries.

Output: Binary classification label (malicious/benign) and classification probability score.

Scoring recipe

def compute_metrics(predictions, labels):
    tp = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 1)
    fp = sum(1 for p, l in zip(predictions, labels) if p == 1 and l == 0)
    fn = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 1)
    tn = sum(1 for p, l in zip(predictions, labels) if p == 0 and l == 0)
    accuracy = (tp + tn) / (tp + fp + fn + tn)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    return accuracy, f1, fpr

Common pitfalls

• The dataset is heavily imbalanced due to concurrent benign system processes; the authors explicitly under-sampled benign traces to match malware traces, which may not reflect real-world deployment distributions.
• Detection latency is highly dependent on the sliding window size (100 vs 500); smaller windows reduce wait time but slightly lower F1 scores, creating a trade-off between speed and accuracy.
• GPU acceleration is assumed for the DL model in deployment, but the paper notes GPUs are not yet widespread on end-user devices, making CPU-only latency a critical practical constraint.

Evidence (verbatim from paper)

Our metrics for model performance were accuracy, precision, recall, F1 score, and false positive (FP) rate.

Citation

@misc{sun2017propedeutica,
  title={Learning Fast and Slow: PROPEDEUTICA for Real-time Malware Detection},
  author={Sun et al. (2017)},
  year={2017},
  note={arXiv:1712.01145}
}

• arXiv: 1712.01145