H2seqrec Eval

h2seqrec-eval

Hyperbolic Hypergraphs for Sequential Recommendation — Li et al. (2021) (arXiv:2108.08134, 2021)

What this evaluates

Evaluates sequential recommendation models on predicting the next item a user will interact with, capturing temporal dynamics and handling sparse user-item interactions.

Datasets

• AMT — total 59495; splits: train (-1), val (-1), test (-1)
• Goodreads — total 1084781; splits: train (-1), val (-1), test (-1)

Metrics

• HR@K (primary) — range: [0, 1]
  • Hit Ratio at rank K: binary indicator (1 if ground truth item appears in top-K predicted list, 0 otherwise). Evaluated at K={1,5,10,20}.
• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank K: sum of graded relevance (1 for hit) discounted by log2(rank+1), normalized by ideal DCG. Evaluated at K={1,5,10,20}.

Input / output format

Input: User's historical interaction sequence (ordered list of item IDs).

Output: Ranked list of candidate items (top-K predictions or full ranking over 101/501 items including sampled negatives).

Scoring recipe

def compute_metrics(predictions, ground_truth, k=10):
    top_k = predictions[:k]
    hr = 1.0 if ground_truth in top_k else 0.0
    ndcg = 0.0
    if ground_truth in top_k:
        rank = top_k.index(ground_truth) + 1
        ndcg = 1.0 / math.log2(rank + 1)
    return hr, ndcg

Common pitfalls

• Negative sampling size (100 vs 500) drastically changes absolute metric values; results are not directly comparable across different sampling settings.
• Datasets use time-based chronological splits (last interactions in a specific year are validation/test), not random user/item splits.
• Evaluation is performed over a sampled candidate set (101 or 501 items) rather than the full item catalog due to computational constraints.

Evidence (verbatim from paper)

Our proposed method focuses on recommending next item, and therefore we use Top K Hit Ratio (HR@K) and Top K Normalized Discounted Cumulative Gain (NDCG@K) as our evaluation metrics. We choose K={1,5,10,20} in the baseline comparison experiment. ... in our experiment, we randomly choose {100,500} negative samples and rank {101,501} items to calculate the HR@K and NDCG@K scores.

Citation

@misc{li2021hyperbolic,
  title={Hyperbolic Hypergraphs for Sequential Recommendation},
  author={Li et al. (2021)},
  year={2021},
  note={arXiv:2108.08134}
}

• arXiv: 2108.08134