Behavioral Fraud Patterns Eval

behavioral-fraud-patterns-eval

Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals — Sajja (2026) (arXiv:2604.13125, 2026)

What this evaluates

This benchmark evaluates whether synthetic tabular data generators preserve complex behavioral fraud patterns beyond simple statistical fidelity. It specifically probes temporal inter-event timing, burst-like transaction sequences, shared-infrastructure graph motifs across accounts, and velocity-rule trigger rates.

Datasets

• IEEE-CIS Fraud Detection — total 590540; splits: train (472432), test (118108)
• Amazon Fraud Dataset — total 151112; splits: test (151112)

Metrics

• P1: IET W1 (primary) — range: seconds
  • Wasserstein-1 distance ($W_1$) between real and synthetic inter-event timing distributions for fraud transactions. Computed on a 50/50 real-data split to establish a baseline.
• P1: within-entity autocorrelation gap — range: dimensionless
  • Absolute difference in autocorrelation of inter-event times within the same entity between real and synthetic data.
• P2: active lifetime W1 — range: seconds
  • Wasserstein-1 distance between real and synthetic active lifetime distributions (time from first to last transaction) for fraud entities.
• P2: burst length W1 — range: dimensionless
  • Wasserstein-1 distance between real and synthetic burst length distributions (transactions within a 5-minute window) for fraud entities.
• P3: fan-out W1 — range: dimensionless
  • Wasserstein-1 distance between real and synthetic fan-out distributions (number of unique devices/IPs per entity) for fraud entities.
• P4: mean velocity-rule trigger rate gap — range: dimensionless
  • Absolute difference in the mean rate of velocity-rule triggers between real and synthetic transaction sequences.

Input / output format

Input: Real tabular transaction records (48 curated behavioral columns) used to train generative models; evaluation receives synthetic tabular rows with externally assigned pseudo-entity IDs.

Output: Synthetic tabular rows matching the 48-column schema (TransactionDT, TransactionAmt, isFraud, card4, C1–C14, D1–D15, M1–M9, addr1, addr2, dist1, dist2, P_emaildomain, R_emaildomain), with pseudo-entity group labels assigned post-generation.

Scoring recipe

def compute_degradation_ratio(predictions, gold, pattern):
    gold_baseline = compute_pattern_metric(gold, pattern)
    synth_score = compute_pattern_metric(predictions, pattern)
    return gold_baseline / synth_score

Common pitfalls

• Row-independent generators cannot natively produce entity identifiers, requiring external pseudo-entity assignment that artificially inflates behavioral fidelity scores.
• Excluding high-cardinality entity columns (e.g., card1) and opaque risk scores (V-columns) during training isolates behavioral coherence but masks downstream utility and scalability failures.
• Unconditional sampling on VAE/GAN architectures causes minority-class (fraud) collapse, necessitating conditional sampling to restore realistic fraud rates before evaluation.

Evidence (verbatim from paper)

We evaluate P1, P2, and P4 on this dataset; it is the richest publicly available source of within-entity transaction sequences. We evaluate P3 (shared-infrastructure graph motifs) exclusively on this dataset; the device and IP co-occurrence structure yields high attribute fan-out for fraud-linked entities, providing a strong signal for graph motif evaluation.

Citation

@misc{sajja2026synthetictabular,
  title={Synthetic Tabular Generators Fail to Preserve Behavioral Fraud Patterns: A Benchmark on Temporal, Velocity, and Multi-Account Signals},
  author={Sajja (2026)},
  year={2026},
  note={arXiv:2604.13125}
}

• arXiv: 2604.13125