Xfraud Fraud Detection Eval

xfraud-fraud-detection-eval

xFraud: Explainable Fraud Transaction Detection — Rao et al. (2020) (arXiv:2011.12193, 2020)

What this evaluates

Evaluates the ability of graph neural networks to detect fraudulent transactions in large-scale, highly imbalanced e-commerce transaction graphs. It probes model performance under extreme class imbalance and measures the trade-off between detection accuracy, inference speed, and scalability across different graph sizes and distributed settings.

Datasets

• eBay-xlarge — total ?; splits: test (-1); repo https://github.com/eBay/xFraud
• eBay-large — total ?; splits: test (-1); repo https://github.com/eBay/xFraud
• eBay-small — total ?; splits: test (-1); repo https://github.com/eBay/xFraud

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• Inference time — range: seconds
  • Wall-clock time required to process a batch of 640 nodes during the testing phase.
• Training time — range: seconds
  • Wall-clock time required to complete one full training epoch.

Input / output format

Input: Heterogeneous transaction graphs containing nodes (e.g., transactions, addresses, payment tokens) and edges, with node features ranging from 114 to 480 dimensions depending on the dataset subset.

Output: Binary classification prediction per node/transaction (fraudulent vs. benign), along with continuous probability scores used to compute ROC and precision-recall curves.

Scoring recipe

def compute_auc(y_true, y_scores):
    # y_true: binary labels (1=fraud, 0=benign)
    # y_scores: predicted probabilities for fraud
    fpr, tpr, _ = roc_curve(y_true, y_scores)
    return auc(fpr, tpr)

Common pitfalls

• The evaluation uses pre-filtered and down-sampled benign transactions, which may not reflect the raw production data distribution.
• Distributed training on 16 machines yields lower AUC than 8 machines due to restricted neighbor fields, highlighting a scalability-accuracy trade-off that readers might overlook.
• Extreme class imbalance (~4% fraud) makes accuracy misleading; AUC and precision-recall curves are necessary for proper assessment.

Evidence (verbatim from paper)

From Table 3, our detector+, achieves the best AUC (averaged across seeds) using 8 machines w.r.t. GEM and GAT. In terms of training efficiency, xFraud detector+ takes only slightly longer time per epoch compared to GEM in an 8-machine setting.

Citation

@misc{rao2020xfraud,
  title={xFraud: Explainable Fraud Transaction Detection},
  author={Rao et al. (2020)},
  year={2020},
  note={arXiv:2011.12193}
}

• arXiv: 2011.12193