Poirot Alignment Eval

poirot-alignment-eval

POIROT: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Hunting — Miljerdi et al. (2019) (arXiv:1910.00056, 2019)

What this evaluates

Evaluates the ability to detect cyber attack campaigns by aligning threat intelligence query graphs with system provenance graphs derived from kernel audit logs. It probes structural pattern matching, causal dependency reasoning, and robustness against malware mutations and benign system noise.

Datasets

• DARPA TC Dataset — total ?; splits: test (-1)
• Public Malware Reports — total ?; splits: test (-1)

Metrics

• alignment score (primary) — range: [0, 1]
  • A float value representing the goodness of structural alignment between a threat intelligence query graph and a system provenance graph. Computed iteratively by matching nodes and edges based on causal dependencies and information flows, normalized to [0, 1].

Input / output format

Input: A query graph ($G_q$) extracted from threat intelligence reports describing attack behavior, and a provenance graph ($G_p$) constructed from kernel audit logs of the target system.

Output: An alignment score (float) and the set of matched nodes/edges between $G_q$ and $G_p$. Detection is confirmed if the score exceeds a threshold of 1/3.

Scoring recipe

def compute_alignment_score(query_graph, provenance_graph):
    # Iteratively align nodes starting from seeds with fewest candidates
    alignment = iterative_node_alignment(query_graph, provenance_graph)
    # Calculate structural overlap score based on matched nodes/edges
    score = calculate_graph_overlap_score(alignment)
    return score

# Detection triggers if score >= 1/3
if compute_alignment_score(G_q, G_p) >= 1/3:
    flag_as_detected()

Common pitfalls

• The threshold $C_{thr}=3$ corresponds to a score threshold of 1/3, not 3.
• The metric measures structural graph alignment, not simple IOC (hash/process name) matching.
• Scores < 1.0 do not necessarily mean failure; they reflect minor causal dependency differences between reports and actual execution.

Evidence (verbatim from paper)

In each iteration, an alignment is constructed, and its alignment score is compared with the threshold value, which is set to $\frac{1}{3}$. ... After finding alignment with a score bypassing the threshold, we manually analyzed all the matched attack subgraphs to confirm that they were correctly pinpointing the actual attacks present in the query graphs.

Citation

@misc{miljerdi2019poirot,
  title={POIROT: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Hunting},
  author={Miljerdi et al. (2019)},
  year={2019},
  note={arXiv:1910.00056}
}

• arXiv: 1910.00056