Cfdb Eval

cfdb-eval

A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation — Jing et al. (2024) (arXiv:2404.14746, 2024)

What this evaluates

This benchmark evaluates machine learning models' ability to detect fraudulent customer activity by analyzing aggregated behavioral patterns and transaction features at the customer level. It probes anomaly detection and risk profiling capabilities on synthetic, privacy-compliant financial data with highly imbalanced class distributions.

Datasets

• CFDB (Customer-level Fraud Detection Benchmark) — total ?; splits: train (-1), test (-1)

Metrics

• Precision — range: [0, 1]
  • Measures the accuracy of positive predictions, i.e., the proportion of predicted fraudulent transactions that were actually fraudulent.
• Recall — range: [0, 1]
  • Also known as sensitivity, it measures the ability of the model to detect all relevant instances, i.e., the proportion of actual fraudulent transactions that were correctly identified by the model.
• Accuracy — range: [0, 1]
  • Provides a general indication of the model's ability to correctly label both fraudulent and non-fraudulent transactions.
• AUC — range: [0, 1]
  • Represents the area under the ROC curve and provides an aggregate measure of performance across all possible classification thresholds.
• F1 Score (primary) — range: [0, 1]
  • The harmonic mean of precision and recall, providing a single score that balances both the concerns of precision and recall in one number.

Input / output format

Input: Customer-level aggregated profiles containing behavioral patterns, transaction features, and network structures derived from underlying transaction-level datasets.

Output: Binary classification label indicating whether a customer profile is fraudulent or legitimate, along with predicted probabilities for threshold-independent metrics.

Scoring recipe

def compute_metrics(y_true, y_pred, y_prob=None):
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 1)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t == 0 and p == 1)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t == 1 and p == 0)
    tn = len(y_true) - tp - fp - fn
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    accuracy = (tp + tn) / len(y_true)
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    auc = roc_auc_score(y_true, y_prob) if y_prob is not None else None
    return {'precision': precision, 'recall': recall, 'accuracy': accuracy, 'f1': f1, 'auc': auc}

Common pitfalls

• The dataset is highly imbalanced, but the authors explicitly state they did not use oversampling methods like SMOTE, which may limit model performance.
• Accuracy is reported but noted as potentially misleading due to class imbalance, so relying solely on it can overstate model effectiveness.
• All models use default hyperparameters without tuning, which may disadvantage more complex architectures like Neural Networks.

Evidence (verbatim from paper)

To assess the performance of each model on the CFDB, we employed a variety of evaluation metrics that provide a comprehensive view of each model's effectiveness in detecting fraudulent transactions: Precision: Measures the accuracy of positive predictions, i.e., the proportion of predicted fraudulent transactions that were actually fraudulent. Recall: Also known as sensitivity, it measures the ability of the model to detect all relevant instances, i.e., the proportion of actual fraudulent transactions that were correctly identified by the model. F1 Score: The harmonic mean of precision and recall, providing a single score that balances both the concerns of precision and recall in one number.

Citation

@misc{jing2024cfdb,
  title={A Customer Level Fraudulent Activity Detection Benchmark for Enhancing Machine Learning Model Research and Evaluation},
  author={Jing et al. (2024)},
  year={2024},
  note={arXiv:2404.14746}
}

• arXiv: 2404.14746