sec-gfd-eval
Revisiting Graph-Based Fraud Detection in Sight of Heterophily and Spectrum — Fan Xu et al. (2023) (arXiv:2312.06441, 2023)
What this evaluates
Evaluates graph neural networks for fraud detection on real-world transaction and review graphs. It specifically probes a model's robustness to severe class imbalance and heterophily, where connected nodes often belong to different classes.
Datasets
Metrics
F1-macro (primary) — range: [0, 100] percent
- Weighted average of F1 scores across all classes (fraud and normal). Reported as a percentage.
AUC (primary) — range: [0, 100] percent
- Area under the Receiver Operating Characteristic (ROC) curve, measuring the trade-off between true positive and false positive rates. Reported as a percentage.
Input / output format
Input: Graph-structured data containing node features, adjacency matrix, and binary node labels (normal vs. fraudulent).
Output: Binary classification label or anomaly score for each node.
Scoring recipe
def compute_metrics(y_true, y_pred):
f1_macro = f1_score(y_true, y_pred, average='macro') * 100
auc = roc_auc_score(y_true, y_pred) * 100
return f1_macro, auc
Common pitfalls
- The train/val/test split ratio is fixed at 0.4/0.2/0.4, which differs from the standard 0.6/0.2/0.2 used in many node classification benchmarks.
- Datasets are highly imbalanced (3–14% anomalies), making accuracy an unreliable metric; F1-macro and AUC must be used.
- Heterophily is prevalent, meaning standard low-pass GNNs (e.g., GCN) are expected to underperform compared to band-pass or high-pass variants.
Evidence (verbatim from paper)
As graph anomaly detection poses a class-imbalanced classification problem, this paper utilizes two widely adopted metrics: F1-macro and AUC. F1-macro considers the weighted average of F1 scores across multiple classes, and AUC is the area under the ROC Curve.
Citation
@misc{xu2023revisiting,
title={Revisiting Graph-Based Fraud Detection in Sight of Heterophily and Spectrum},
author={Fan Xu et al. (2023)},
year={2023},
note={arXiv:2312.06441}
}
1---2name: sec-gfd-eval3description: Evaluates graph neural networks for fraud detection on real-world transaction and review graphs. It specifically probes a model's robustness to severe class imbalance and heterophily, where connected nodes often belong to different classes. Use when the user wants to benchmark on Amazon, YelpChi, T-Finance, T-Social, or asks about evaluating this task. Reports F1-macro, AUC.4---56# sec-gfd-eval78> Revisiting Graph-Based Fraud Detection in Sight of Heterophily and Spectrum — Fan Xu et al. (2023) (arXiv:2312.06441, 2023)910## What this evaluates1112Evaluates graph neural networks for fraud detection on real-world transaction and review graphs. It specifically probes a model's robustness to severe class imbalance and heterophily, where connected nodes often belong to different classes.1314## Datasets1516- **Amazon** — total 11944; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD17- **YelpChi** — total 45954; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD18- **T-Finance** — total 39357; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD19- **T-Social** — total 5781065; splits: train (-1), val (-1), test (-1); repo https://github.com/Sunxkissed/SEC-GFD2021## Metrics2223- `F1-macro` **(primary)** — range: [0, 100] percent24 - Weighted average of F1 scores across all classes (fraud and normal). Reported as a percentage.25- `AUC` **(primary)** — range: [0, 100] percent26 - Area under the Receiver Operating Characteristic (ROC) curve, measuring the trade-off between true positive and false positive rates. Reported as a percentage.2728## Input / output format2930**Input**: Graph-structured data containing node features, adjacency matrix, and binary node labels (normal vs. fraudulent).3132**Output**: Binary classification label or anomaly score for each node.3334## Scoring recipe3536```python37def compute_metrics(y_true, y_pred):38 f1_macro = f1_score(y_true, y_pred, average='macro') * 10039 auc = roc_auc_score(y_true, y_pred) * 10040 return f1_macro, auc41```4243## Common pitfalls4445- The train/val/test split ratio is fixed at 0.4/0.2/0.4, which differs from the standard 0.6/0.2/0.2 used in many node classification benchmarks.46- Datasets are highly imbalanced (3–14% anomalies), making accuracy an unreliable metric; F1-macro and AUC must be used.47- Heterophily is prevalent, meaning standard low-pass GNNs (e.g., GCN) are expected to underperform compared to band-pass or high-pass variants.4849## Evidence (verbatim from paper)5051> As graph anomaly detection poses a class-imbalanced classification problem, this paper utilizes two widely adopted metrics: F1-macro and AUC. F1-macro considers the weighted average of F1 scores across multiple classes, and AUC is the area under the ROC Curve.5253## Citation5455```bibtex56@misc{xu2023revisiting,57 title={Revisiting Graph-Based Fraud Detection in Sight of Heterophily and Spectrum},58 author={Fan Xu et al. (2023)},59 year={2023},60 note={arXiv:2312.06441}61}62```6364- arXiv: 2312.06441
qhjqhj00