Ieee Cis Fraud Eval

ieee-cis-fraud-eval

Shapley Value-Guided Adaptive Ensemble Learning for Explainable Financial Fraud Detection with U.S. Regulatory Compliance Validation — Uddin et al. (2026) (arXiv:2604.14231, 2026)

What this evaluates

Evaluates binary financial fraud detection performance across diverse model architectures (LSTM, Transformer, XGBoost, GNN, and ensembles) on highly imbalanced transaction data. Probes threshold-independent discrimination (AUC-ROC, PR-AUC) and threshold-dependent detection accuracy (F1, Precision, Recall, MCC) under stratified cross-validation and temporal holdout conditions.

Datasets

• IEEE-CIS Financial Fraud Detection Dataset — total ?; splits: train (442905), test (147635), held-out (118108)

Metrics

• PR-AUC (primary) — range: [0, 1]
  • Area under the Precision-Recall curve, computed over all classification thresholds. Designed for highly imbalanced binary classification where the positive class (fraud) constitutes only 3.5% of samples.
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across thresholds.
• F1 — range: [0, 1]
  • Harmonic mean of Precision and Recall. Reported at each model's individually optimized F1-optimal threshold (τ*), not a fixed global threshold.
• MCC — range: [-1, 1]
  • Matthews Correlation Coefficient, a balanced measure of binary classification quality that accounts for true/false positives and negatives.

Input / output format

Input: Tabular transaction features (431 raw features) with sequential account histories for RNNs/Transformers, and transaction graph structures for GNNs. Binary fraud label with a 3.5% positive rate.

Output: Predicted fraud probability per transaction. Hard predictions derived at model-specific F1-optimal threshold τ* for reporting Precision, Recall, F1, and MCC.

Scoring recipe

def evaluate(y_true, y_prob):
    # Primary metric: threshold-independent
    pr_auc = average_precision_score(y_true, y_prob)
    # Threshold-dependent metrics: optimize per model
    best_thresh = optimize_threshold(y_true, y_prob, metric='f1')
    y_pred = (y_prob >= best_thresh).astype(int)
    f1 = f1_score(y_true, y_pred)
    mcc = matthews_corrcoef(y_true, y_pred)
    return {'PR-AUC': pr_auc, 'F1': f1, 'MCC': mcc}

Common pitfalls

• PR-AUC is explicitly designated as the primary metric due to severe class imbalance (3.5% fraud rate); relying solely on AUC-ROC or Accuracy can be misleading.
• F1, Precision, and Recall are reported at each model's individually optimized F1-threshold (τ*), not a fixed global threshold, making direct cross-model threshold comparisons invalid.
• SMOTE-Tomek oversampling is applied strictly within training folds during cross-validation; failing to replicate this fold-wise resampling will cause data leakage and inflated metrics.

Evidence (verbatim from paper)

PR-AUC is the primary metric for this imbalanced classification task. All F1, Precision, Recall, and MCC values reported at each model's F1-optimal threshold τ*; GNN-GraphSAGE τ* = 0.86 reflects skewed fraud probability outputs in sparse graph structures.

Citation

@misc{uddin2026shapley,
  title={Shapley Value-Guided Adaptive Ensemble Learning for Explainable Financial Fraud Detection with U.S. Regulatory Compliance Validation},
  author={Uddin et al. (2026)},
  year={2026},
  note={arXiv:2604.14231}
}

• arXiv: 2604.14231