# Semignn Alipay Eval

> Evaluates a semi-supervised graph neural network's ability to predict user loan defaults and classify user occupations using multiview graph data (social ties, app usage, nicks, addresses) on a large-scale financial platform dataset. Use when the user wants to benchmark on Alipay, or asks about evaluating this task. Reports AUC.

- Skill: `qhjqhj00/semignn-alipay-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/semignn-alipay-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/semignn-alipay-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/semignn-alipay-eval

---


# semignn-alipay-eval

> A Semi-supervised Graph Attentive Network for Financial Fraud Detection — Wang et al. (2020) (arXiv:2003.01171, 2020)

## What this evaluates

Evaluates a semi-supervised graph neural network's ability to predict user loan defaults and classify user occupations using multiview graph data (social ties, app usage, nicks, addresses) on a large-scale financial platform dataset.

## Datasets

- **Alipay** — total ?; splits: train (-1), val (-1), test (-1)

## Metrics

- `AUC` **(primary)** — range: [0, 1]
  - Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to distinguish between default and non-default users across all classification thresholds.
- `KS` — range: [0, 1]
  - Kolmogorov-Smirnov statistic. Measures risk differentiation by finding the maximum difference between the cumulative distribution functions of positive (default) and negative (non-default) classes.
- `F1-score` — range: [0, 1]
  - Harmonic mean of precision and recall. Used for multi-class occupation prediction.
- `Precision` — range: [0, 1]
  - Ratio of correctly predicted positive occupations to all predicted positives.
- `Recall` — range: [0, 1]
  - Ratio of correctly predicted positive occupations to all actual positives.
- `Top-1% Precision` — range: [0, 1]
  - Precision calculated only on the top 1% of users ranked by predicted risk/occupation probability, focusing on high-confidence predictions for financial risk control.

## Input / output format

**Input**: Multiview graph data per user: user-relation graph (social ties), user-app graph (app login frequency), user-nick graph (nick words), user-address graph (address words). Node features are derived from pretrained embeddings of these views. Labels are provided for a subset of users (default/non-default or occupation).

**Output**: Binary prediction for user default (default vs non-default) or multi-class prediction for user occupation.

## Scoring recipe

```python
def compute_auc(y_true, y_score):
    return sklearn.metrics.roc_auc_score(y_true, y_score)

def compute_ks(y_true, y_score):
    order = np.argsort(y_score)[::-1]
    y_true_sorted = y_true[order]
    pos_cum = np.cumsum(y_true_sorted)
    neg_cum = np.cumsum(1 - y_true_sorted)
    total_pos = pos_cum[-1]
    total_neg = neg_cum[-1]
    return np.max(np.abs(pos_cum/total_pos - neg_cum/total_neg))
```

## Common pitfalls

- The dataset is highly imbalanced (only 5% labeled as default), making accuracy misleading; AUC and KS are preferred for risk differentiation.
- Evaluation splits (50/20/30) apply only to the 4M labeled users, while >100M unlabeled users are used for semi-supervised training but excluded from metric calculation.
- Top-1% Precision is a domain-specific metric for financial risk control, not a standard ML metric, and may be misinterpreted as standard precision.

## Evidence (verbatim from paper)

> Commonly, we use AUC as the evaluation metric. Specifically, financial scenario also concerns about the KS, which is a metric to measure the risk differentiation of the model.

## Citation

```bibtex
@misc{wang2020semignn,
  title={A Semi-supervised Graph Attentive Network for Financial Fraud Detection},
  author={Wang et al. (2020)},
  year={2020},
  note={arXiv:2003.01171}
}
```

- arXiv: 2003.01171