Magic Apt Detection Eval

magic-apt-detection-eval

MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning — Zian Jia et al. (2023) (arXiv:2310.09831, 2023)

What this evaluates

Evaluates the ability of a self-supervised graph representation learning model to detect Advanced Persistent Threats (APTs) in system audit logs. It probes multi-granularity anomaly detection (batched log-level and system entity-level) under a strict unsupervised setting where only benign data is available for training.

Datasets

• StreamSpot — total ?; splits: train (400), test (200)
• Unicorn Wget — total ?; splits: train (100), test (50)
• DARPA Engagement 3 — total ?; splits: train (-1), test (-1)

Metrics

• Precision (primary) — range: [0, 1]
  • Ratio of correctly predicted malicious instances to all instances predicted as malicious. Formula: TP / (TP + FP).
• Recall — range: [0, 1]
  • Ratio of correctly predicted malicious instances to all actual malicious instances. Formula: TP / (TP + FN).
• FPR — range: [0, 1]
  • False Positive Rate: Ratio of incorrectly predicted malicious instances to all actual benign instances. Formula: FP / (FP + TN).
• F1-Score — range: [0, 1]
  • Harmonic mean of Precision and Recall. Formula: 2 * (Precision * Recall) / (Precision + Recall).
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic curve, measuring the model's ability to distinguish between classes across all thresholds.

Input / output format

Input: System audit logs (system calls, file operations, network connections) aggregated into provenance graphs. Inputs are provided as batches of logs for log-level detection or individual system entities for entity-level detection.

Output: A detection score or binary classification label (benign/malicious) for each input log batch or system entity.

Scoring recipe

tp = sum(pred == 1 and gold == 1)
fp = sum(pred == 1 and gold == 0)
fn = sum(pred == 0 and gold == 1)
tn = sum(pred == 0 and gold == 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
auc = compute_auc(gold, scores) # from ROC curve

Common pitfalls

• Training exclusively on benign data forces the model to learn normal behavior distributions rather than attack signatures, making it sensitive to concept drift and novel benign patterns.
• Highly imbalanced test sets (e.g., DARPA E3) can mask poor performance if accuracy is reported; FPR and Recall are the critical metrics here.
• Detection granularity varies significantly: batched log-level detection aggregates system calls, while entity-level detection isolates individual processes/files, leading to different performance characteristics (e.g., hard to detect passive files/libraries).

Evidence (verbatim from paper)

MAGIC yields low FPR (average 0.15%) with large training data. This is because MAGIC models benign system behaviors with self-supervised embeddings, allowing it to effectively handle unseen system entities.

Citation

@misc{jia2023magic,
  title={MAGIC: Detecting Advanced Persistent Threats via Masked Graph Representation Learning},
  author={Zian Jia et al. (2023)},
  year={2023},
  note={arXiv:2310.09831}
}

• arXiv: 2310.09831