Malware Static Reachability Eval

malware-static-reachability-eval

Mining Malware Specifications through Static Reachability Analysis — Macedo et al. (2013) (arXiv:1312.4814, 2013)

What this evaluates

Evaluates a static analysis tool's ability to detect malware by extracting system call data flow trees and matching them against a learned tree automaton. It probes the model's capability to generalize semantic malware signatures from a small training set to a larger, unseen test set while avoiding false positives on benign software.

Datasets

• VX Heavens & Windows XP Benign — total 1426; splits: train (193), test_malware (983), test_benign (250)

Metrics

• detection_rate (primary) — range: [0, 1]
  • True Positives / Total Malware Samples in Test Set. Reports the fraction of unseen malware binaries correctly identified as malicious.
• false_positive_rate — range: [0, 1]
  • False Positives / Total Benign Samples in Test Set. Reports the fraction of benign Windows XP binaries incorrectly flagged as malware.

Input / output format

Input: Windows executable binary files (PE format).

Output: Binary classification label: 'malware' if any extracted System Call Data Flow Tree (SCDT) subtree is recognized by the learned tree automaton, otherwise 'benign'.

Scoring recipe

def evaluate(predictions, gold_labels):
    tp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'malware' and g == 'malware')
    fp = sum(1 for p, g in zip(predictions, gold_labels) if p == 'malware' and g == 'benign')
    total_malware = sum(1 for g in gold_labels if g == 'malware')
    total_benign = sum(1 for g in gold_labels if g == 'benign')
    detection_rate = tp / total_malware if total_malware > 0 else 0.0
    fpr = fp / total_benign if total_benign > 0 else 0.0
    return {'detection_rate': detection_rate, 'false_positive_rate': fpr}

Common pitfalls

• The train/test split is arbitrary and not stratified by malware family, which may affect generalization claims across different threat types.
• Static analysis relies on external tools (Jakstab, IDA Pro) and approximations for stack values, which can lead to overapproximated trees and missed detections if the oracle is imprecise.
• Detection is binary and based on subtree matching; benign programs with similar system calls but different control/data flow may be misclassified if the automaton is too permissive.

Evidence (verbatim from paper)

We were able to detect 983 malware files using the malicious trees inferred from 193 malware files, and show that benign programs are benign, thus a 0% false positive rate.

Citation

@misc{macedo2013mining,
  title={Mining Malware Specifications through Static Reachability Analysis},
  author={Macedo et al. (2013)},
  year={2013},
  note={arXiv:1312.4814}
}

• arXiv: 1312.4814