Gadbench Eval

gadbench-eval

GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection — Tang et al. (2023) (arXiv:2306.12251, 2023)

What this evaluates

Evaluates supervised graph anomaly detection capabilities on static attributed graphs. It benchmarks models across transductive and inductive settings, homogeneous and heterogeneous graph structures, and compares traditional tree ensembles with neighbor aggregation against standard and specialized GNNs.

Datasets

• Reddit — total ?; splits: transductive (-1), inductive (-1)
• Weibo — total ?; splits: transductive (-1), inductive (-1)
• Amazon — total ?; splits: transductive (-1), inductive (-1)
• Yelp — total ?; splits: transductive (-1), inductive (-1)
• T-Fin — total ?; splits: transductive (-1), inductive (-1)
• Ellip — total ?; splits: transductive (-1), inductive (-1)
• Tolo — total ?; splits: transductive (-1), inductive (-1)
• Quest — total ?; splits: transductive (-1), inductive (-1)
• DGraph — total ?; splits: transductive (-1), inductive (-1)
• T-Social — total ?; splits: transductive (-1), inductive (-1)

Metrics

• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Computed by plotting the true positive rate against the false positive rate at various classification thresholds and calculating the area under this curve.
• AUPRC (primary) — range: [0, 1]
  • Area under the Precision-Recall curve. Computed by plotting precision against recall at various thresholds and calculating the area under this curve. Preferred for highly imbalanced anomaly detection tasks.
• Rec@K — range: [0, 1]
  • Recall at top-K predicted anomalies. Calculated as the number of true anomalies among the top-K highest-scoring predictions divided by the total number of true anomalies in the dataset.

Input / output format

Input: Graph adjacency structure (edges), node feature vectors, and node labels (for supervised or semi-supervised training).

Output: Anomaly score or probability for each node in the graph.

Scoring recipe

import numpy as np
from sklearn.metrics import roc_auc_score, average_precision_score

def compute_metrics(predictions, gold, k=10):
    auroc = roc_auc_score(gold, predictions)
    auprc = average_precision_score(gold, predictions)
    top_k_idx = np.argsort(predictions)[::-1][:k]
    rec_at_k = np.sum(gold[top_k_idx]) / np.sum(gold)
    return {'AUROC': auroc, 'AUPRC': auprc, f'Rec@{k}': rec_at_k}

Common pitfalls

• Default hyperparameters often severely underperform compared to optimally tuned ones, especially for GNNs, making un-tuned comparisons misleading.
• Inductive settings (where test node features/structure are hidden during training) drastically reduce performance compared to transductive settings, particularly on temporal datasets.
• Tree ensembles with simple neighbor aggregation frequently outperform specialized GNNs, challenging the assumption that graph neural networks are inherently superior for graph anomaly detection.

Evidence (verbatim from paper)

The performance gap becomes particularly significant in the fully-supervised setting, i.e., XGB-Graph surpasses BWGNN—the best GNN model in this setting—by an absolute average improvement of 2.0% on AUROC, 12.9% on AUPRC, and 9.8% on Rec@K.

Citation

@misc{tang2023gadbench,
  title={GADBench: Revisiting and Benchmarking Supervised Graph Anomaly Detection},
  author={Tang et al. (2023)},
  year={2023},
  note={arXiv:2306.12251}
}

• arXiv: 2306.12251