Bwgnn Anomaly Detection Eval

bwgnn-anomaly-detection-eval

Rethinking Graph Neural Networks for Anomaly Detection — Tang et al. (2022) (arXiv:2205.15508, 2022)

What this evaluates

Evaluates graph neural networks and baselines for node anomaly detection in heterogeneous and homogeneous graphs. It probes the model's ability to identify fraudulent or anomalous accounts/users based on node features and graph structure under both supervised and semi-supervised settings.

Datasets

• YelpChi — total ?; splits: train (-1), val (-1), test (-1)
• Amazon — total ?; splits: train (-1), val (-1), test (-1)
• T-Finance — total ?; splits: train (-1), val (-1), test (-1)
• T-Social — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• F1-macro (primary) — range: percent
  • The unweighted mean of the F1-score of two classes (normal and anomaly), which neglects the imbalance ratio between normal and anomaly labels.
• AUC — range: [0, 1]
  • The area under the Receiver Operating Characteristic (ROC) Curve, measuring the model's ability to discriminate between normal and anomalous nodes across all classification thresholds.

Input / output format

Input: Graph data containing node features and edge connections. Multi-relational graphs are processed either by treating all edges as a single homogeneous type or by performing separate propagation per relation.

Output: Binary anomaly label (normal vs. anomaly) or continuous anomaly score for each node.

Scoring recipe

def compute_metrics(y_true, y_scores, threshold=0.5):
    y_pred = (y_scores >= threshold).astype(int)
    f1_normal = f1_score(y_true, y_pred, pos_label=0)
    f1_anomaly = f1_score(y_true, y_pred, pos_label=1)
    f1_macro = (f1_normal + f1_anomaly) / 2
    auc = roc_auc_score(y_true, y_scores)
    return f1_macro, auc

Common pitfalls

• F1-macro is used specifically to ignore class imbalance between normal and anomaly nodes, unlike standard accuracy or weighted F1.
• Multi-relational graphs (YelpChi, Amazon) require explicit handling strategies (homogeneous vs. heterogeneous propagation) that drastically affect results.
• Semi-supervised settings use extremely low training ratios (1% or 0.01%), making performance highly sensitive to the exact number of labeled anomalies.

Evidence (verbatim from paper)

Metrics. We choose two widely used metrics to measure the performance of all the methods, namely F1-macro and AUC. F1-macro is the unweighted mean of the F1-score of two classes, which neglects the imbalance ratio between normal and anomaly labels. AUC (Davis & Goadrich, 2006) is the area under the ROC Curve.

Citation

@misc{tang2022rethinking,
  title={Rethinking Graph Neural Networks for Anomaly Detection},
  author={Tang et al. (2022)},
  year={2022},
  note={arXiv:2205.15508}
}

• arXiv: 2205.15508