Yelpchi Fraud Detection Eval

yelpchi-fraud-detection-eval

Alleviating the Inconsistency Problem of Applying Graph Neural Network to Fraud Detection — Zhiwei Liu et al. (arXiv:2005.00625, 2020)

What this evaluates

This evaluation probes a graph neural network's ability to detect fraudulent or spam reviews within a multi-relational graph structure. It measures classification robustness against structural and semantic inconsistencies by training on varying fractions of labeled data and testing on the remainder.

Datasets

• YelpChi — total 45954; splits: train (-1), test (-1)

Metrics

• F1-score (primary) — range: [0, 1]
  • Harmonic mean of precision and recall: 2 * (precision * recall) / (precision + recall). Measures overall classification performance on the imbalanced spam/legitimate review task.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. Measures the model's ability to rank spam reviews higher than legitimate ones across all classification thresholds.

Input / output format

Input: Graph-structured data where nodes represent reviews, each initialized with a 100-dimensional Word2Vec feature vector. Edges encode three relations: same user (R-U-R), same product & rating (R-S-R), and same product & month (R-T-R). Each node has a binary ground-truth label (spam or legitimate).

Output: Binary classification label (spam/legitimate) or a continuous confidence score for each review node.

Scoring recipe

def compute_metrics(y_true, y_pred_proba):
    y_pred = (y_pred_proba >= 0.5).astype(int)
    tp = ((y_pred == 1) & (y_true == 1)).sum()
    fp = ((y_pred == 1) & (y_true == 0)).sum()
    fn = ((y_pred == 0) & (y_true == 1)).sum()
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0
    auc = roc_auc_score(y_true, y_pred_proba)
    return {'F1': f1, 'AUC': auc}

Common pitfalls

• The evaluation uses variable training data percentages (40%, 60%, 80%) rather than a fixed train/val/test split, requiring strict random seed control for reproducibility.
• The graph construction explicitly filters products with >800 reviews and builds three specific multi-relational edges, deviating from standard single-relation node classification benchmarks.
• AUC is computed on prediction probabilities while F1 uses a fixed 0.5 threshold; reporting them interchangeably without specifying the threshold causes metric misalignment.

Evidence (verbatim from paper)

We use F1-score to measure the overall classification performance and AUC to measure the performance of identifying spam reviews.

Citation

@misc{liu2020alleviating,
  title={Alleviating the Inconsistency Problem of Applying Graph Neural Network to Fraud Detection},
  author={Zhiwei Liu et al.},
  year={2020},
  note={arXiv:2005.00625}
}

• arXiv: 2005.00625