Credit Card Fraud Detection Eval

credit-card-fraud-detection-eval

Transformer-Based Financial Fraud Detection with Cloud-Optimized Real-Time Streaming — Deng et al. (2025) (arXiv:2501.19267, 2025)

What this evaluates

Evaluates a model's ability to detect fraudulent credit card transactions in a streaming context by learning topological and sequential patterns from transaction graphs without manual feature engineering.

Datasets

• Credit Card Transaction Dataset (Feb-Sep) — total 141861; splits: train (-1), test (-1)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision: AP = (1/N) * Σ P(n) for n=1 to N, measuring precision at each threshold across the dataset.
• AUC — range: [0, 1]
  • Area Under the ROC Curve: AUC = ∫ TPR(FPR) d(FPR) from 0 to 1, measuring the overall ability to distinguish between fraudulent and normal transactions.

Input / output format

Input: Original credit card transaction records (merchant number, card number, amount, etc.) converted into a dynamic transaction graph where nodes represent transactions/entities and edges represent relationships.

Output: Fraud probability score for each transaction.

Scoring recipe

def compute_metrics(predictions, labels):
    # predictions: list of fraud probabilities
    # labels: list of binary fraud labels (1=fraud, 0=normal)
    # Calculate AP (Average Precision)
    ap = sum(precision_at_threshold(t) for t in thresholds) / len(thresholds)
    # Calculate AUC (Area Under ROC Curve)
    auc = trapezoidal_integration(tpr, fpr)
    return ap, auc

Common pitfalls

• Temporal split: training on Feb-Jun and testing on Jul-Sep, so results are not independent and identically distributed across months.
• Imbalanced data: requires negative sampling on normal transactions, which can skew precision/recall if not handled consistently.
• Evaluation reported monthly (July, August, September) rather than as a single aggregate metric, making cross-month comparison sensitive to seasonal fraud patterns.

Evidence (verbatim from paper)

The experimental data set in this paper is based on partial credit card transaction records from February 1 to September 30 of a certain year. ... During the training process, data from February 1 to June 30 were selected as the training set and data from July 1 to September 30 as the test set. ... In this experiment, AUC (Area Under the ROC Curve) and AP (Average Precision) were used as the measurement indexes of credit card fraudulent transaction detection tasks.

Citation

@misc{deng2025transformerfinancialfraud,
  title={Transformer-Based Financial Fraud Detection with Cloud-Optimized Real-Time Streaming},
  author={Deng et al. (2025)},
  year={2025},
  note={arXiv:2501.19267}
}

• arXiv: 2501.19267