Dataco Fraud Detection Eval

dataco-fraud-detection-eval

Semi-Supervised Supply Chain Fraud Detection with Unsupervised Pre-Filtering — Moradi et al. (2025) (arXiv:2508.06574, 2025)

What this evaluates

Evaluates semi-supervised anomaly detection models for supply chain fraud under severe class imbalance and limited label availability. Probes the ability to leverage unsupervised pre-filtering and self-training to improve precision, recall, and F1-score while maintaining low false positive rates.

Datasets

• DataCo Smart Supply Chain Dataset — total 180519; splits: 10-fold cross-validation (-1)

Metrics

• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Optimized for highly imbalanced fraud detection.
• AUC-ROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across classification thresholds.
• AUC-PR — range: [0, 1]
  • Area under the Precision-Recall curve, preferred for evaluating performance on highly imbalanced datasets where the positive class is rare.
• FPR — range: [0, 1]
  • False Positive Rate: FP / (FP + TN). The protocol targets keeping this below 3.0%.

Input / output format

Input: 52 raw features per transaction (numerical, categorical, temporal) representing supply chain operations across multiple countries.

Output: Binary fraud label (fraud vs. legitimate) or anomaly score/confidence for each transaction.

Scoring recipe

def compute_metrics(y_true, y_pred):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 0)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    fpr = fp / (fp + tn) if (fp + tn) > 0 else 0
    return {'F1-Score': f1, 'Precision': precision, 'Recall': recall, 'FPR': fpr}

Common pitfalls

• The Isolation Forest contamination factor was set to 0.05 (5%), not the dataset's actual 1.5% fraud rate, to capture borderline anomalies.
• Only 10% of the training portion is labeled per fold, with the remaining 90% treated as unlabeled for self-training, simulating realistic label scarcity.
• Feature selection strictly removed 3 features with >80% missing values and 2 highly correlated features (Pearson r>0.95), resulting in exactly 47 final features.

Evidence (verbatim from paper)

Given the class imbalance inherent in fraud detection, we employed multiple evaluation metrics (Precision, Recall, F1-Score, AUC-ROC, AUC-PR, FPR) with statistical significance testing via Wilcoxon signed-rank test. We compared our approach against six baselines... Within each fold of the 10-fold cross-validation, the training portion (approximately 162,467 transactions per fold) was split such that 10% served as labeled data ($\mathcal{D}{L}$ with 16,247 samples per fold), while the remaining 90% ( 146,220 transactions per fold) formed the unlabeled set $\mathcal{D}{U}$.

Citation

@misc{moradi2025semisupervised,
  title={Semi-Supervised Supply Chain Fraud Detection with Unsupervised Pre-Filtering},
  author={Moradi et al. (2025)},
  year={2025},
  note={arXiv:2508.06574}
}

• arXiv: 2508.06574