Transfer Fraud Detection Eval

transfer-fraud-detection-eval

Instance-Dependent Cost-Sensitive Learning for Detecting Transfer Fraud — Höppner et al. (2020) (arXiv:2005.02488, 2020)

What this evaluates

Evaluates machine learning models for detecting fraudulent bank transfers by optimizing instance-dependent cost-sensitive objectives. It probes the model's ability to minimize financial losses and maximize expected savings under highly imbalanced transaction data.

Datasets

• Credit Card Transaction Data — total 284807; splits: full (284807)
• Bank data set — total 31763; splits: full (31763)

Metrics

• Expected Savings (primary) — range: other
  • Net financial gain maximized by minimizing Average Expected Cost (AEC). Calculated as the difference between gains from correctly identified frauds and costs incurred from false positives.
• Savings — range: other
  • Total financial savings achieved by the model's binary decisions compared to a baseline, computed after applying instance-dependent cost-related thresholds.
• Precision — range: [0, 1]
  • Ratio of true positive fraud predictions to all positive predictions.
• Recall — range: [0, 1]
  • Ratio of true positive fraud predictions to all actual fraud cases.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall.

Input / output format

Input: Numerical and categorical features representing bank transactions (e.g., PCA-transformed features V1-V28, Time, Amount, or proprietary bank features).

Output: Binary classification decision (fraud or not) derived from predicted probabilities using an instance-dependent cost-related threshold.

Scoring recipe

tp = sum(y_true == 1 & y_pred == 1)
fp = sum(y_true == 0 & y_pred == 1)
fn = sum(y_true == 1 & y_pred == 0)
precision = tp / (tp + fp) if (tp + fp) > 0 else 0
recall = tp / (tp + fn) if (tp + fn) > 0 else 0
f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
# Expected Savings & Savings are computed by applying instance-dependent cost thresholds to predicted probabilities, converting them to binary decisions, and calculating net financial gain/loss per transaction.

Common pitfalls

• Folds must be stratified by both fraud label and transaction amount category (low/middle/high) to maintain distribution balance across the 5x2-fold cross-validation replications.
• Using standard fixed thresholds instead of instance-dependent cost-related thresholds leads to suboptimal financial performance despite potentially higher F1 scores.
• Optimizing purely for accuracy-related metrics (Precision, Recall, F1) can result in higher overall financial costs compared to cost-sensitive objectives.

Evidence (verbatim from paper)

All methods are evaluated using Savings, Expected Savings, Precision, Recall and $F_{1}$ measure where we use the instance-dependent thresholds. For each data set, we perform 5 replications of two-fold cross validation. To keep the analysis of the data sets manageable, we only consider main effects and we do not include interactions of any degree.

Citation

@misc{hoppner2020instance,
  title={Instance-Dependent Cost-Sensitive Learning for Detecting Transfer Fraud},
  author={Höppner et al. (2020)},
  year={2020},
  note={arXiv:2005.02488}
}

• arXiv: 2005.02488