Birdnest Fraud Detection Eval

birdnest-fraud-detection-eval

BIRDNEST: Bayesian Inference for Ratings-Fraud Detection — Hooi et al. (2015) (arXiv:1511.06030, 2015)

What this evaluates

This benchmark evaluates a model's ability to detect fraudulent user accounts in e-commerce and app review platforms by analyzing their rating patterns and temporal behavior. It probes whether the model can identify users who exhibit extreme rating biases or bursty posting times indicative of coordinated spam or defamation campaigns.

Datasets

• Flipkart — total 3300000; splits: test (-1)
• SWM — total 1100000; splits: test (-1)

Metrics

• precision@k (primary) — range: [0, 1]
  • Precision@k is calculated as the number of true fraudulent users found in the top-k ranked list divided by k. The paper reports this metric for k values up to 250.

Input / output format

Input: User-level rating histories including product IDs, star ratings (1-5), and timestamps.

Output: A continuous suspiciousness score (NEST) for each user, used to rank users from most to least suspicious.

Scoring recipe

def precision_at_k(rankings, gold_fraud_ids, k):
    top_k_users = rankings[:k]
    true_positives = sum(1 for u in top_k_users if u in gold_fraud_ids)
    return true_positives / k

Common pitfalls

• Ground truth labels are only available for the Flipkart dataset via private investigation by the platform, making independent replication impossible.
• The SWM dataset evaluation is purely qualitative (case studies), so no quantitative metrics are reported for it.
• Precision@k is only evaluated for small k (up to 250), ignoring performance at larger cutoffs or overall ranking quality.

Evidence (verbatim from paper)

Figure 1b shows the algorithm's precision at $k$ : for various values of $k$ up to 250: note that precision for the most suspicious users is very high: e.g. precision of 1.0 for the first 50 users.

Citation

@misc{hooi2015birdnest,
  title={BIRDNEST: Bayesian Inference for Ratings-Fraud Detection},
  author={Hooi et al. (2015)},
  year={2015},
  note={arXiv:1511.06030}
}

• arXiv: 1511.06030