Grecx Eval

grecx-eval

GRecX: An Efficient and Unified Benchmark for GNN-based Recommendation — Cai et al. (2021) (arXiv:2111.10342, 2021)

What this evaluates

Evaluates the recommendation accuracy and inference efficiency of GNN-based and matrix factorization models on real-world interaction datasets. It specifically probes how well models perform under standardized evaluation conditions that account for varying negative sampling strategies and representation dimensions.

Datasets

• yelp2018 — total ?; splits: test (-1); repo https://github.com/maenzhier/GRecX
• gowalla — total ?; splits: test (-1); repo https://github.com/maenzhier/GRecX
• amazon-book — total ?; splits: test (-1); repo https://github.com/maenzhier/GRecX

Metrics

• NDCG@20 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20. Computes the weighted sum of relevance scores at each position up to 20, normalized by the ideal DCG for the given ground truth.
• GRMF-X — range: [0, 1]
  • A unified NDCG@20 metric designed to fairly compare models trained with different negative sampling sizes (e.g., 1 vs 800) by adjusting the evaluation procedure to account for sampling strategy differences.

Input / output format

Input: User and item interaction graphs or matrices, along with user/item IDs for ranking.

Output: A ranked list of candidate items for each user, evaluated at the top-20 positions.

Scoring recipe

def compute_ndcg_at_20(relevant_items, predicted_items):
    dcg = 0.0
    for i, item in enumerate(predicted_items[:20]):
        if item in relevant_items:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_items), 20)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Comparing models trained with vastly different negative sampling sizes (1 vs 800) without using a unified metric like GRMF-X leads to unfair performance assessments.
• Ignoring representation dimensionality differences (e.g., NGCF concatenates layer outputs resulting in 256D vs standard 64D) can skew comparisons unless explicitly normalized or controlled.
• Mixing loss functions (BCE vs BPR) during training without reporting them separately makes direct metric comparison invalid.

Evidence (verbatim from paper)

Here we employ a widely-used metrics NDCG@20 and our new metrics GRMF-X (GRMF-NDCG@20) for evaluation. As mentioned in Section 2.1.1, for other hyper-parameters, we carefully tune these hyper-parameters and report the performance with them. In terms of number of negative samples, we set it to 1 and 800 to align with the original experimental settings of NGCF (one negative sample), LightGCN (one negative sample) and UltraGCNbase (800 negative samples).

Citation

@misc{cai2021grecx,
  title={GRecX: An Efficient and Unified Benchmark for GNN-based Recommendation},
  author={Cai et al. (2021)},
  year={2021},
  note={arXiv:2111.10342}
}

• arXiv: 2111.10342