Youtube Implicit Rec Eval

youtube-implicit-rec-eval

A Generic Coordinate Descent Framework for Learning from Implicit Feedback — Bayer et al. (2016) (arXiv:1611.04666, 2016)

What this evaluates

Evaluates recommender systems on implicit feedback datasets, testing their ability to rank relevant items for users. It probes model versatility across cold-start, offline, and instant recommendation scenarios using side information and sequential context features.

Datasets

• YouTube Implicit Feedback Subset — total ?; splits: train (-1), test (-1)

Metrics

• NDCG@100 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 100. Computes the sum of graded relevance scores discounted by log2(rank+1), normalized by the ideal DCG.
• Recall@100 — range: [0, 1]
  • Fraction of relevant items in the ground truth that appear in the top 100 recommended items.

Input / output format

Input: User ID, interaction history, and optional side/context features (age, gender, country, device, previously watched video, all previously watched videos, user ID).

Output: Ranked list of top 100 recommended video IDs.

Scoring recipe

def compute_metrics(preds, gold, k=100):
    rel = [1 if item in gold else 0 for item in preds[:k]]
    dcg = sum(r / math.log2(i + 2) for i, r in enumerate(rel))
    ideal_rel = sorted(rel, reverse=True)
    idcg = sum(r / math.log2(i + 2) for i, r in enumerate(ideal_rel))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    recall = sum(rel) / len(gold) if gold else 0.0
    return ndcg, recall

Common pitfalls

• The paper reports relative improvements over a Popularity baseline rather than absolute metric values.
• The dataset is a proprietary YouTube subset; exact train/test split sizes and the tuning holdout are not publicly disclosed.
• Different evaluation scenarios (Cold-Start, Offline, Instant) use fundamentally different train/eval partitioning strategies.

Evidence (verbatim from paper)

We measure the recall and NDCG for the top 100 returned videos. Note that we report relative improvements over the Popularity recommender.

Citation

@misc{bayer2016generic,
  title={A Generic Coordinate Descent Framework for Learning from Implicit Feedback},
  author={Bayer et al. (2016)},
  year={2016},
  note={arXiv:1611.04666}
}

• arXiv: 1611.04666