Fraudster Group Detection Eval

fraudster-group-detection-eval

Spatio-Temporal Graph Representation Learning for Fraudster Group Detection — Shehnepoor et al. (2022) (arXiv:2201.02621, 2022)

What this evaluates

Evaluates a model's ability to detect fraudulent reviewer groups by analyzing spatio-temporal co-review patterns. It probes the model's capacity to distinguish genuine groups from coordinated fraudster groups using graph representation learning and temporal modeling.

Datasets

• Yelp — total 9952; splits: train (-1), test (-1)
• Amazon — total 2194; splits: train (-1), test (-1)

Metrics

• Precision — range: [0, 1]
  • Precision = TP / (TP + FP), where TP is the number of true positive samples and FP is the number of false positive samples.
• Recall — range: [0, 1]
  • Recall = TP / (TP + FN), where FN is the number of false negative samples.
• F1-value (primary) — range: [0, 1]
  • F1-value = 2 * precision * recall / (precision + recall).

Input / output format

Input: Candidate reviewer groups represented as spatio-temporal graphs derived from co-review interactions within 28-day windows, along with reviewer/item metadata and ratings.

Output: Binary classification label for each candidate group: 'genuine' or 'fraudster'.

Scoring recipe

def calculate_metrics(predictions, gold):
    tp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 1)
    fp = sum(1 for p, g in zip(predictions, gold) if p == 1 and g == 0)
    fn = sum(1 for p, g in zip(predictions, gold) if p == 0 and g == 1)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return precision, recall, f1

Common pitfalls

• The evaluation is performed at the group level, not the individual reviewer level.
• Temporal modeling performance degrades on the Amazon dataset due to sparse group data in early time windows.
• Clustering is an optional post-processing step; omitting it changes the precision/recall trade-off by retaining outlier reviewers.

Evidence (verbatim from paper)

For evaluation, we used 80% of the data for training and 20% for testing. We used three well-known metrics to evaluate the performance of the proposed approach. First, precision: Precision = TP/(TP+FP) where TP is the number of true positive samples and FP is the number of false positive samples. We also use recall: Recall = TP/(TP+FN) where FN is the number of false negative samples. Finally, we also use the F1-value: F1-value = 2precisionrecall/(precision+recall)

Citation

@misc{shehnepoor2022spatiotemporal,
  title={Spatio-Temporal Graph Representation Learning for Fraudster Group Detection},
  author={Shehnepoor et al. (2022)},
  year={2022},
  note={arXiv:2201.02621}
}

• arXiv: 2201.02621