# Ieee Cis Fraud Eval

> Evaluates binary financial fraud detection performance across diverse model architectures (LSTM, Transformer, XGBoost, GNN, and ensembles) on highly imbalanced transaction data. Probes threshold-independent discrimination (AUC-ROC, PR-AUC) and threshold-dependent detection accuracy (F1, Precision, Recall, MCC) under stratified cross-validation and temporal holdout conditions. Use when the user wants to benchmark on IEEE-CIS Financial Fraud Detection Dataset, or asks about evaluating this task. Reports PR-AUC.

- Skill: `qhjqhj00/ieee-cis-fraud-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/ieee-cis-fraud-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ieee-cis-fraud-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/ieee-cis-fraud-eval

---


# ieee-cis-fraud-eval

> Shapley Value-Guided Adaptive Ensemble Learning for Explainable Financial Fraud Detection with U.S. Regulatory Compliance Validation — Uddin et al. (2026) (arXiv:2604.14231, 2026)

## What this evaluates

Evaluates binary financial fraud detection performance across diverse model architectures (LSTM, Transformer, XGBoost, GNN, and ensembles) on highly imbalanced transaction data. Probes threshold-independent discrimination (AUC-ROC, PR-AUC) and threshold-dependent detection accuracy (F1, Precision, Recall, MCC) under stratified cross-validation and temporal holdout conditions.

## Datasets

- **IEEE-CIS Financial Fraud Detection Dataset** — total ?; splits: train (442905), test (147635), held-out (118108)

## Metrics

- `PR-AUC` **(primary)** — range: [0, 1]
  - Area under the Precision-Recall curve, computed over all classification thresholds. Designed for highly imbalanced binary classification where the positive class (fraud) constitutes only 3.5% of samples.
- `AUC-ROC` — range: [0, 1]
  - Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across thresholds.
- `F1` — range: [0, 1]
  - Harmonic mean of Precision and Recall. Reported at each model's individually optimized F1-optimal threshold (τ*), not a fixed global threshold.
- `MCC` — range: [-1, 1]
  - Matthews Correlation Coefficient, a balanced measure of binary classification quality that accounts for true/false positives and negatives.

## Input / output format

**Input**: Tabular transaction features (431 raw features) with sequential account histories for RNNs/Transformers, and transaction graph structures for GNNs. Binary fraud label with a 3.5% positive rate.

**Output**: Predicted fraud probability per transaction. Hard predictions derived at model-specific F1-optimal threshold τ* for reporting Precision, Recall, F1, and MCC.

## Scoring recipe

```python
def evaluate(y_true, y_prob):
    # Primary metric: threshold-independent
    pr_auc = average_precision_score(y_true, y_prob)
    # Threshold-dependent metrics: optimize per model
    best_thresh = optimize_threshold(y_true, y_prob, metric='f1')
    y_pred = (y_prob >= best_thresh).astype(int)
    f1 = f1_score(y_true, y_pred)
    mcc = matthews_corrcoef(y_true, y_pred)
    return {'PR-AUC': pr_auc, 'F1': f1, 'MCC': mcc}
```

## Common pitfalls

- PR-AUC is explicitly designated as the primary metric due to severe class imbalance (3.5% fraud rate); relying solely on AUC-ROC or Accuracy can be misleading.
- F1, Precision, and Recall are reported at each model's individually optimized F1-threshold (τ*), not a fixed global threshold, making direct cross-model threshold comparisons invalid.
- SMOTE-Tomek oversampling is applied strictly within training folds during cross-validation; failing to replicate this fold-wise resampling will cause data leakage and inflated metrics.

## Evidence (verbatim from paper)

> PR-AUC is the primary metric for this imbalanced classification task. All F1, Precision, Recall, and MCC values reported at each model's F1-optimal threshold τ*; GNN-GraphSAGE τ* = 0.86 reflects skewed fraud probability outputs in sparse graph structures.

## Citation

```bibtex
@misc{uddin2026shapley,
  title={Shapley Value-Guided Adaptive Ensemble Learning for Explainable Financial Fraud Detection with U.S. Regulatory Compliance Validation},
  author={Uddin et al. (2026)},
  year={2026},
  note={arXiv:2604.14231}
}
```

- arXiv: 2604.14231