Banksim Fraud Detection Eval

banksim-fraud-detection-eval

Financial Fraud Detection: A Comparative Study of Quantum Machine Learning Models — Innan et al. (2023) (arXiv:2308.05237, 2023)

What this evaluates

Evaluates the ability of quantum machine learning models to classify synthetic financial transactions as fraudulent or benign based on demographic, merchant, and transactional features. It probes the models' capacity to handle imbalanced binary classification tasks and extract discriminative patterns from tabular financial data.

Datasets

• BankSim — total 594643; splits: train (-1), test (-1)

Metrics

• F1 score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Optimized for imbalanced binary classification where false positives and false negatives carry significant cost.

Input / output format

Input: Tabular instance with four selected features: Age (integer), Gender (LabelEncoder encoded), Category (LabelEncoder encoded), and Amount (float). Target variable: Fraud (0 for benign, 1 for fraudulent).

Output: Binary class label: 0 (benign) or 1 (fraudulent).

Scoring recipe

def compute_f1(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    return 2 * (precision * recall) / (precision + recall) if (precision + recall) > 0 else 0

Common pitfalls

• The raw dataset is highly imbalanced (7,200 fraud vs 587,443 benign); the authors artificially balanced it to 200 records, which may not reflect real-world deployment conditions or class distribution.
• Feature selection (PCA + logical analysis) was performed on the full dataset, but the final models were trained on a tiny 200-record subset, increasing the risk of overfitting and reducing generalizability.
• Evaluation relies on a classical quantum circuit simulator (Qiskit Aer Qasm-Simulator) rather than actual NISQ hardware, so hardware noise, decoherence, and gate errors are not captured in the reported metrics.

Evidence (verbatim from paper)

The original dataset was loaded, and specific subsets were extracted to create a balanced dataset containing 200 records with 100 instances of fraudulent and non-fraudulent transactions. ... The dataset was split into training and testing sets using the train_test_split function from scikit-learn to facilitate the model training process. ... Following these preprocessing steps and dividing the dataset into training and testing sets, the data was ready for the subsequent model training and evaluation processes. The QSVC achieves the highest F1 score (0.98) across fraud and non-fraud classes...

Citation

@misc{innan2023financial,
  title={Financial Fraud Detection: A Comparative Study of Quantum Machine Learning Models},
  author={Innan et al. (2023)},
  year={2023},
  note={arXiv:2308.05237}
}

• arXiv: 2308.05237