Semignn Alipay Eval

semignn-alipay-eval

A Semi-supervised Graph Attentive Network for Financial Fraud Detection — Wang et al. (2020) (arXiv:2003.01171, 2020)

What this evaluates

Evaluates a semi-supervised graph neural network's ability to predict user loan defaults and classify user occupations using multiview graph data (social ties, app usage, nicks, addresses) on a large-scale financial platform dataset.

Datasets

• Alipay — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between default and non-default users across all classification thresholds.
• KS — range: [0, 1]
  • Kolmogorov-Smirnov statistic. Measures risk differentiation by finding the maximum difference between the cumulative distribution functions of positive (default) and negative (non-default) classes.
• F1-score — range: [0, 1]
  • Harmonic mean of precision and recall. Used for multi-class occupation prediction.
• Precision — range: [0, 1]
  • Ratio of correctly predicted positive occupations to all predicted positives.
• Recall — range: [0, 1]
  • Ratio of correctly predicted positive occupations to all actual positives.
• Top-1% Precision — range: [0, 1]
  • Precision calculated only on the top 1% of users ranked by predicted risk/occupation probability, focusing on high-confidence predictions for financial risk control.

Input / output format

Input: Multiview graph data per user: user-relation graph (social ties), user-app graph (app login frequency), user-nick graph (nick words), user-address graph (address words). Node features are derived from pretrained embeddings of these views. Labels are provided for a subset of users (default/non-default or occupation).

Output: Binary prediction for user default (default vs non-default) or multi-class prediction for user occupation.

Scoring recipe

def compute_auc(y_true, y_score):
    return sklearn.metrics.roc_auc_score(y_true, y_score)

def compute_ks(y_true, y_score):
    order = np.argsort(y_score)[::-1]
    y_true_sorted = y_true[order]
    pos_cum = np.cumsum(y_true_sorted)
    neg_cum = np.cumsum(1 - y_true_sorted)
    total_pos = pos_cum[-1]
    total_neg = neg_cum[-1]
    return np.max(np.abs(pos_cum/total_pos - neg_cum/total_neg))

Common pitfalls

• The dataset is highly imbalanced (only 5% labeled as default), making accuracy misleading; AUC and KS are preferred for risk differentiation.
• Evaluation splits (50/20/30) apply only to the 4M labeled users, while >100M unlabeled users are used for semi-supervised training but excluded from metric calculation.
• Top-1% Precision is a domain-specific metric for financial risk control, not a standard ML metric, and may be misinterpreted as standard precision.

Evidence (verbatim from paper)

Commonly, we use AUC as the evaluation metric. Specifically, financial scenario also concerns about the KS, which is a metric to measure the risk differentiation of the model.

Citation

@misc{wang2020semignn,
  title={A Semi-supervised Graph Attentive Network for Financial Fraud Detection},
  author={Wang et al. (2020)},
  year={2020},
  note={arXiv:2003.01171}
}

• arXiv: 2003.01171