Travelfraudbench Eval

travelfraudbench-eval

TRAVELFRAUDBENCH: A Configurable Evaluation Framework for GNN Fraud Ring Detection in Travel Networks — Sajja (2026) (arXiv:2604.21093, 2026)

What this evaluates

Evaluates graph neural networks and tabular baselines on detecting fraudulent user rings in travel booking networks. It probes the models' ability to classify individual fraud accounts and recover entire fraud ring structures using heterogeneous graph topology and co-occurrence signals.

Datasets

• TravelFraudBench (TFG) — total 10000; splits: train (6000), val (2000), test (2000); repo https://github.com/bhavana3/travel-fraud-graphs

Metrics

• AUC-ROC (primary) — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the trade-off between true positive rate and false positive rate across all classification thresholds.
• Average Precision (AP) — range: [0, 1]
  • Area under the Precision-Recall curve, summarizing performance across all thresholds with higher weight on positive class performance.
• F1 — range: [0, 1]
  • Harmonic mean of precision and recall, computed at the threshold that maximizes F1 on the validation set.
• Ring recall at threshold — range: [0, 1]
  • Fraction of test-set fraud rings where ≥80% of members are assigned a fraud probability score above 0.5.

Input / output format

Input: Per user node: 10-dimensional tabular features (e.g., account age, booking velocity, device/IP counts) and graph structure (either a projected homogeneous user-user co-occurrence graph or a full heterogeneous graph with 9 node types and 12 edge relations).

Output: Per user node: fraud probability score (softmax output). Per ring: binary recovery status determined by whether ≥80% of its members exceed the 0.5 prediction threshold.

Scoring recipe

# Task 1: Node Classification
y_true = [1 if user in fraud_ring else 0 for user in test_users]
y_scores = model.predict_proba(test_users)
auc_roc = compute_auc_roc(y_true, y_scores)
ap = compute_average_precision(y_true, y_scores)
best_thresh = argmax_f1(y_val_true, y_val_scores)
f1 = compute_f1(y_true, y_scores > best_thresh)

# Task 2: Ring Recovery
rings = get_test_rings()
recovered = 0
for ring in rings:
    members = ring.members
    scores = model.predict_proba(members)
    if sum(s > 0.5 for s in scores) >= 0.8 * len(members):
        recovered += 1
ring_recall = recovered / len(rings)

Common pitfalls

• The MLP baseline has access to fewer features than GNNs (feature-access asymmetry), which can inflate the perceived graph-structural advantage if not controlled via ablation.
• Ring recovery requires near-perfect member prediction (≥80% threshold); partial hits do not count, making the task operationally strict and sensitive to threshold selection.
• The benchmark uses a ring-based split to prevent transductive leakage, meaning standard random node splits will artificially inflate performance and invalidate results.

Evidence (verbatim from paper)

We evaluate on two tasks: Task 1: Binary Node Classification. Classify each user node as fraud (1) or legitimate (0). We use the standard 60/20/20 train/validation/test split, stratified by ring membership. Reported metrics: AUC-ROC, Average Precision (AP), and F1 at the threshold maximizing F1 on the validation set. Task 2: Ring Recovery. Given model output scores, identify fraud ring memberships. We define a ring as recovered if ≥80% of its members are assigned a score above the decision threshold (threshold = 0.5 on the model’s softmax fraud probability). We report ring recall at threshold: the fraction of test-set rings meeting this criterion.

Citation

@misc{sajja2026travelfraudbench,
  title={TRAVELFRAUDBENCH: A Configurable Evaluation Framework for GNN Fraud Ring Detection in Travel Networks},
  author={Sajja (2026)},
  year={2026},
  note={arXiv:2604.21093}
}

• arXiv: 2604.21093