Outlier Detection Eval

outlier-detection-eval

Credit Card Fraud Detection in e-Commerce: An Outlier Detection Approach — Porwale et al. (2018) (arXiv:1811.02196, 2018)

What this evaluates

Evaluates the ability of an outlier detection algorithm to identify anomalous data points in highly imbalanced datasets without prior knowledge of fraud patterns. It probes consistency estimation and ensemble clustering robustness across varying feature spaces and class distributions.

Datasets

• Satimage-2 — total 5803; splits: test (-1)
• Thyroid — total 3772; splits: test (-1)
• Credit Card Fraud Detection — total 284807; splits: test (-1); repo https://www.kaggle.com/mlg-ulb/creditcardfraud

Metrics

• AUPRC (primary) — range: [0, 1]
  • Area under the Precision-Recall curve. Precision = TP / (TP + FP), Recall = TP / (TP + FN). Computed by sweeping thresholds on the consistency score and integrating the resulting PR curve.
• AUROC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve. Plots True Positive Rate (TP / (TP + FN)) against False Positive Rate (FP / (FP + TN)) across thresholds.

Input / output format

Input: Numerical feature vectors per data point. Credit Card dataset: 30 PCA-transformed features + Time + Amount. Satimage-2: 36 spectral band features. Thyroid: 6 real-valued attributes.

Output: A continuous consistency score per data point (weighted centroid signature similarity). Outliers are flagged by low scores; binary predictions are derived via thresholding for metric computation.

Scoring recipe

def compute_auprc(scores, labels):
    thresholds = np.linspace(scores.min(), scores.max(), 100)
    precisions, recalls = [], []
    for t in thresholds:
        preds = (scores <= t).astype(int)
        tp = np.sum((preds == 1) & (labels == 1))
        fp = np.sum((preds == 1) & (labels == 0))
        fn = np.sum((preds == 0) & (labels == 1))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        precisions.append(prec)
        recalls.append(rec)
    return np.trapz(precisions, recalls)

Common pitfalls

• Using AUROC as the sole evaluation metric for highly imbalanced outlier detection, which can be misleading due to the overwhelming number of true negatives.
• Running the k-means ensemble only once without reporting variance, as initialization and data ordering significantly affect consistency scores.
• Attempting to quantitatively compare against baselines that only publish visual PR/AUC curves without numerical results, making exact reproduction impossible.

Evidence (verbatim from paper)

We are going to use area under the precision recall curve (AUPRC) as our primary metric to evaluate the performance of the proposed and baseline methods. Note that majority of the work in the outlier detection literature has used AUROC as their metric for evaluation but we argue that AUPRC should also be reported because algorithms that optimize the area under the ROC curve are not guaranteed to optimize the area under the PR curve. Both metrics collectively present a complete picture of the performance of the algorithm because of their focus on different classes.

Citation

@misc{porwale2018credit,
  title={Credit Card Fraud Detection in e-Commerce: An Outlier Detection Approach},
  author={Porwale et al. (2018)},
  year={2018},
  note={arXiv:1811.02196}
}

• arXiv: 1811.02196