Fraud Detection Eval

fraud-detection-eval

Who Audits the Auditor? Tamper-Proof Fraud Detection with Blockchain-Anchored Explainable ML — Wang (2026) (arXiv:2604.22096, 2026)

What this evaluates

This evaluation probes a machine learning model's ability to accurately detect fraudulent financial transactions in highly imbalanced tabular data, while also measuring the system-level overhead and economic viability of integrating blockchain-based audit trails. It tests both detection accuracy on real-world and synthetic datasets and the practical throughput/latency constraints of on-chain verification workflows.

Datasets

• Kaggle Credit Card Fraud — total 284807; splits: 5-fold stratified CV (-1)
• Enterprise Payment Dataset — total 10000; splits: 5-fold stratified CV (-1)

Metrics

• F1 (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Optimized for imbalanced fraud detection where false positives and false negatives carry significant operational costs.
• PR-AUC (primary) — range: [0, 1]
  • Area under the Precision-Recall curve computed across all classification thresholds. Preferred over ROC-AUC for highly imbalanced datasets (0.17% fraud rate) as it focuses on the positive class performance.
• Precision — range: [0, 1]
  • True Positives / (True Positives + False Positives). Measures the proportion of flagged transactions that are actually fraudulent.
• Recall — range: [0, 1]
  • True Positives / (True Positives + False Negatives). Measures the proportion of actual frauds correctly identified by the model.

Input / output format

Input: Tabular transaction features representing individual payment requests (e.g., amount, vendor history, approval timing, budget utilization).

Output: Binary fraud classification (or fraud probability score) and a SHAP explanation listing the top-5 contributing features for the prediction.

Scoring recipe

def compute_metrics(y_true, y_pred_proba, threshold=0.5):
    y_pred = (y_pred_proba >= threshold).astype(int)
    tp = np.sum((y_pred == 1) & (y_true == 1))
    fp = np.sum((y_pred == 1) & (y_true == 0))
    fn = np.sum((y_pred == 0) & (y_true == 1))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    # PR-AUC computed across thresholds
    precisions, recalls, _ = precision_recall_curve(y_true, y_pred_proba)
    pr_auc = np.trapz(precisions, recalls)
    return {'precision': prec, 'recall': rec, 'f1': f1, 'pr_auc': pr_auc}

Common pitfalls

• PR-AUC is frequently confused with ROC-AUC; the paper explicitly uses PR-AUC due to the extreme class imbalance (0.17% fraud rate), making ROC-AUC misleadingly optimistic.
• Blockchain confirmation latency (3–5 seconds on Polygon PoS) is often overlooked as a system bottleneck, reducing end-to-end throughput to ~200 tx/min compared to ML-only batch throughput (45,000/sec).
• The synthetic Enterprise Payment Dataset lacks real-world enterprise features and is explicitly acknowledged as a limitation; results should not be generalized to production environments without external validation.

Evidence (verbatim from paper)

Table II presents detection results using 5-fold stratified cross-validation. Our LightGBM-based detector achieves the highest F1 (0.895) and PR-AUC (0.974).

Citation

@misc{wang2026tamperproof,
  title={Who Audits the Auditor? Tamper-Proof Fraud Detection with Blockchain-Anchored Explainable ML},
  author={Wang (2026)},
  year={2026},
  note={arXiv:2604.22096}
}

• arXiv: 2604.22096