Titant Fraud Detection Eval

titant-fraud-detection-eval

TitAnt: Online Real-time Transaction Fraud Detection in Ant Financial — Cao et al. (2019) (arXiv:1906.07407, 2019)

What this evaluates

Evaluates the ability of machine learning models to detect fraudulent financial transactions in real-time using aggregated transaction network features and basic attributes. It probes how well different feature engineering and classification approaches handle severe label imbalance and temporal data splits.

Datasets

• Ant Financial Transaction Dataset — total ?; splits: train (-1), test (-1), network_build (-1)

Metrics

• F1 Score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Used as the headline metric for daily performance comparison across eleven configurations.
• Recall@top 1% — range: [0, 1]
  • Recall calculated on the top 1% of transactions ranked by predicted fraud probability. Measures the classifier's ability to identify the most suspicious fraud cases.

Input / output format

Input: Transaction records represented by 52 basic features, optionally concatenated with 32-dimensional user node embeddings learned from a transaction network (via DeepWalk or Supervised Node2Vec). Features are discretized into bins for rule-based models.

Output: Binary fraud prediction (fraud/non-fraud) or continuous fraud probability score used for ranking and threshold-based evaluation.

Scoring recipe

def compute_f1(preds, gold):
    tp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(preds, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(preds, gold) if p == 0 and g == 1)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0

def compute_rec_top1(scores, gold):
    ranked = np.argsort(-scores)
    top_k = int(len(gold) * 0.01)
    top_indices = ranked[:top_k]
    return sum(1 for i in top_indices if gold[i] == 1) / sum(gold)

Common pitfalls

• Labels are not available in real-time for online testing, so evaluation strictly uses a T+1 offline training/next-day testing split rather than standard random or chronological splits.
• Rule-based models (ID3, C5.0) require data discretization into bins before training, which significantly impacts their performance compared to continuous models.
• Severe label imbalance means supervised embedding methods (S2V) can underperform unsupervised ones (DW) despite using label information.

Evidence (verbatim from paper)

In this section, we empirically evaluate the effectiveness of our proposed system for the transaction fraud detection task. Eleven configurations are tested in Table 1 from April 10 to April 16, where F1 score is chosen as the evaluation metric.

Citation

@misc{cao2019titant,
  title={TitAnt: Online Real-time Transaction Fraud Detection in Ant Financial},
  author={Cao et al. (2019)},
  year={2019},
  note={arXiv:1906.07407}
}

• arXiv: 1906.07407