Kairos Eval

kairos-eval

Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance — Cheng et al. (2023) (arXiv:2308.05034, 2023)

What this evaluates

Evaluates a provenance-based intrusion detection system's ability to identify anomalous system behavior and reconstruct attack footprints from whole-system kernel-level logs. It probes the model's capacity to distinguish between benign and malicious activity in temporal windows without relying on attack signatures.

Datasets

• Manzoor et al. — total 600; splits: train (5), val (120), test (475)
• DARPA-E3-THEIA — total ?; splits: train (-1), val (-1), test (-1)
• DARPA-E3-CADETS — total ?; splits: train (-1), val (-1), test (-1)
• DARPA-E3-ClearScope — total ?; splits: train (-1), val (-1), test (-1)
• DARPA-E5-THEIA — total ?; splits: train (-1), val (-1), test (-1)
• DARPA-E5-CADETS — total ?; splits: train (-1), val (-1), test (-1)
• DARPA-E5-ClearScope — total ?; splits: train (-1), val (-1), test (-1)
• DARPA-OpTC — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Precision — range: [0, 1]
  • Ratio of true positives to all predicted positives: TP / (TP + FP). Computed per 15-minute time window.
• Recall — range: [0, 1]
  • Ratio of true positives to all actual positives: TP / (TP + FN). Computed per 15-minute time window.
• Accuracy — range: [0, 1]
  • Ratio of correct predictions to total predictions: (TP + TN) / (TP + TN + FP + FN). Computed per 15-minute time window.
• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic (ROC) curve, measuring the model's ability to discriminate between benign and attack time windows across all classification thresholds.

Input / output format

Input: Temporal kernel-level provenance graphs (nodes, edges, timestamps) aggregated into 15-minute time windows.

Output: Binary classification label per time window (benign or anomalous/attack), plus a reconstructed compact attack summary graph for investigation.

Scoring recipe

tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
tn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 0)
fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
accuracy = (tp + tn) / (tp + tn + fp + fn)
# AUC computed via ROC curve over time-window predictions

Common pitfalls

• Ground truth often dismisses entities that remain active after an attack, causing the system to flag them as false positives even though they are still compromised.
• Metrics are computed per 15-minute time window, not per individual edge or node, which can mask fine-grained detection errors.
• Concept drift from new benign applications in test data triggers high reconstruction errors, requiring incremental retraining to maintain precision.

Evidence (verbatim from paper)

Table 4 shows the precision, recall, accuracy, and area under ROC curve (AUC) results for all datasets. We compute these metrics based on time windows. As mentioned in §5.1, we manually label each time window in a provenance graph as either benign or attack according to the ground truth. If KAIROS marks a benign time window as anomalous (i.e., if KAIROS mistakenly includes a benign time window in an anomalous queue), we consider the time window to be a false positive (FP). On the other hand, if KAIROS correctly marks an attack time window as anomalous, it is counted as a true positive (TP). False negatives (FN) and true negatives (TN) are calculated in a similar fashion.

Citation

@misc{cheng2023kairos,
  title={Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance},
  author={Cheng et al. (2023)},
  year={2023},
  note={arXiv:2308.05034}
}

• arXiv: 2308.05034