Ncf Implicit Rec Eval

ncf-implicit-rec-eval

Neural Collaborative Filtering — Xiangnan He et al. (arXiv:1708.05031, 2017)

What this evaluates

Evaluates a model's ability to predict implicit user-item interactions and rank relevant items for recommendation. It probes non-linear collaborative filtering capabilities on sparse, implicit feedback datasets by measuring whether the true interacted item appears near the top of a ranked list.

Datasets

• MovieLens — total 1000209; splits: train (-1), test (6040)
• Pinterest — total 1500809; splits: train (-1), test (55187)

Metrics

• HR@10 (primary) — range: [0, 1]
  • Hit Ratio at rank 10. Returns 1.0 if the ground-truth test item is ranked within the top 10 positions of the candidate list, and 0.0 otherwise. Averaged across all test users.
• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes 1/log2(rank+1) if the test item is in the top 10, else 0. Normalized by the ideal DCG (which is 1 when the test item is ranked first). Averaged across all test users.

Input / output format

Input: User ID and Item ID pairs representing implicit interactions (binary 0/1). During evaluation, a user's historical interactions are used to generate a ranked list of candidate items.

Output: A ranked list of items (or predicted relevance scores) for each user, evaluated against a held-out test item.

Scoring recipe

def evaluate(user_history, test_item, sampled_negatives, k=10):
    candidates = sampled_negatives + [test_item]
    scores = model.predict(user_history, candidates)
    ranked = sorted(candidates, key=lambda x: scores[x], reverse=True)
    test_rank = ranked.index(test_item) + 1
    hr = 1.0 if test_rank <= k else 0.0
    ndcg = (1.0 / math.log2(test_rank + 1)) if test_rank <= k else 0.0
    return hr, ndcg
# Average HR and NDCG over all users

Common pitfalls

• The evaluation ranks the test item against only 100 randomly sampled negative items, not the full item catalog. This speeds up computation but means metrics reflect relative ranking within a small candidate set rather than absolute catalog ranking.
• Datasets must be preprocessed to implicit feedback (binary 0/1) and filtered to retain only users with at least 20 interactions before applying the leave-one-out split.
• Training uses a negative sampling ratio of 4 negatives per positive instance, which is distinct from the 100-item evaluation sampling strategy and should not be confused.

Evidence (verbatim from paper)

To evaluate the performance of item recommendation, we adopted the leave-one-out evaluation, which has been widely used in literature [1, 14, 27]. For each user, we held-out her latest interaction as the test set and utilized the remaining data for training. Since it is too time-consuming to rank all items for every user during evaluation, we followed the common strategy [6, 21] that randomly samples 100 items that are not interacted by the user, ranking the test item among the 100 items. The performance of a ranked list is judged by Hit Ratio (HR) and Normalized Discounted Cumulative Gain (NDCG) [11]. Without special mention, we truncated the ranked list at 10 for both metrics.

Citation

@misc{he2017neural,
  title={Neural Collaborative Filtering},
  author={Xiangnan He et al.},
  year={2017},
  note={arXiv:1708.05031}
}

• arXiv: 1708.05031