Lightgcn Rec Eval

lightgcn-rec-eval

LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation — He et al. (2020) (arXiv:2002.02126, 2020)

What this evaluates

Evaluates the ability of graph-based collaborative filtering models to rank relevant items for users based on sparse user-item interaction graphs. It probes how well neighborhood aggregation and embedding smoothing capture latent preferences without relying on node semantic features.

Datasets

• Gowalla — total 1027370; splits: train (-1), test (-1)
• Yelp2018 — total 1561406; splits: train (-1), test (-1)
• Amazon-Book — total 2984108; splits: train (-1), test (-1)

Metrics

• recall@20 (primary) — range: [0, 1]
  • Recall@20 measures the fraction of a user's ground-truth interacted items that appear in the top-20 recommended items. It is averaged across all users in the test set.
• ndcg@20 — range: [0, 1]
  • NDCG@20 measures the quality of the top-20 ranked list by weighting hits by their position using logarithmic discounting, normalized by the ideal DCG. It is averaged across all users in the test set.

Input / output format

Input: User and item identifiers with a bipartite interaction graph representing observed user-item engagements.

Output: A ranked list of candidate items for each user, evaluated at the top-20 cutoff.

Scoring recipe

def compute_metrics(predictions, ground_truth, k=20):
    recalls, ndcgs = [], []
    for u in ground_truth:
        relevant = ground_truth[u]
        ranked = predictions[u][:k]
        hits = sum(1 for item in ranked if item in relevant)
        recalls.append(hits / len(relevant))
        dcg = sum(hits_at_i / math.log2(i + 2) for i, item in enumerate(ranked) if item in relevant)
        idcg = sum(1 / math.log2(i + 2) for i in range(min(len(relevant), k)))
        ndcgs.append(dcg / idcg if idcg > 0 else 0)
    return {'recall@20': sum(recalls)/len(recalls), 'ndcg@20': sum(ndcgs)/len(ndcgs)}

Common pitfalls

• Using a fixed negative sampling set instead of the all-ranking protocol (the paper explicitly states all non-interacted items are candidates).
• Including cold-start items in the test set without filtering, which was a known issue in the original Yelp2018 split requiring a revised version.
• Comparing models trained or evaluated on different train-test splits, as the authors explicitly requested the exact same splits from the NGCF authors for fairness.

Evidence (verbatim from paper)

The evaluation metrics are recall@20 and ndcg@20 computed by the all-ranking protocol — all items that are not interacted by a user are the candidates.

Citation

@misc{he2020lightgcn,
  title={LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation},
  author={He et al. (2020)},
  year={2020},
  note={arXiv:2002.02126}
}

• arXiv: 2002.02126