Dgser Eval

dgser-eval

Dynamic Graph Neural Networks for Sequential Recommendation — Zhang et al. (2021) (arXiv:2104.07368, 2021)

What this evaluates

Evaluates a model's ability to perform sequential next-item recommendation by modeling dynamic collaborative signals and temporal user preferences. It tests how well the system captures high-order item transitions and time-annotated graph structures to predict the next interaction in a user's history.

Datasets

• Amazon-CDs — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Games — total ?; splits: train (-1), val (-1), test (-1)
• Amazon-Beauty — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Measures the quality of the ranked list by discounting the relevance of the ground-truth item based on its position, normalized by the ideal DCG.
• Hit@10 — range: [0, 1]
  • Binary metric indicating whether the ground-truth item appears in the top-10 recommended items.

Input / output format

Input: A user's historical interaction sequence with associated timestamps, used to predict the next item in the sequence.

Output: A ranked list of top-10 candidate items (including the ground-truth item and 100 randomly sampled negatives).

Scoring recipe

# For each test user:
# 1. Sample 100 negative items uniformly at random.
# 2. Combine with ground-truth item to form a candidate set of 101 items.
# 3. Score each candidate using the model and sort descending.
# 4. Find rank of ground-truth item (1-indexed).
# Hit@10 = 1 if rank <= 10 else 0
# DCG@10 = 1 / log2(rank + 1)
# NDCG@10 = DCG@10 / 1 (ideal DCG for single relevant item at rank 1 is 1)
# Average Hit@10 and NDCG@10 over all test samples.

Common pitfalls

• Negative sampling is fixed at 100 items per test sample, which differs from the full-item ranking used in some other recommendation benchmarks.
• Datasets are filtered to only include users and items with at least 5 interactions, which significantly reduces sparsity compared to raw Amazon dumps.
• Sequences are split chronologically (last item=test, second last=val, rest=train), not randomly shuffled.

Evidence (verbatim from paper)

We adopt two widely-used metrics Hit@$K$ and NDCG@$K$, to evaluate all methods. Hit@$K$ indicates the proportion of the ground-truth items among the top@$K$ items, while NDCG@$K$ is position-aware metric, and higher NDCG means target items tend to have more top rank positions. Following [23, 11], for each test sample, we randomly sample 100 negative items, and rank these items with the ground-truth item. We evaluate Hit@$K$ and NDCG@$K$ based on these 101 items. By default, we set $K$=10.

Citation

@misc{zhang2021dgser,
  title={Dynamic Graph Neural Networks for Sequential Recommendation},
  author={Zhang et al. (2021)},
  year={2021},
  note={arXiv:2104.07368}
}

• arXiv: 2104.07368