Ninerrec Eval

ninerrec-eval

NineRec: A Benchmark Dataset Suite for Evaluating Transferable Recommendation — Zhang et al. (2023) (arXiv:2309.07705, 2023)

What this evaluates

This benchmark evaluates transferable recommendation (TransRec) and modality-only recommendation (MoRec) capabilities. It probes whether models can learn to recommend items using only text and image features without relying on item IDs, and how well they transfer across different domains and modalities.

Datasets

• NineRec (source: Bili_500K, targets: Bili_Food, Bili_Dance, Bili_Movie, Bili_Cartoon, Bili_Music, KU, QB, TN, DY) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/westlake-repl/NineRec

Metrics

• H@10 (primary) — range: percent
  • Hit Ratio @K equals 1 if the ground-truth item appears in the top-K predicted items, else 0. Averaged over all users.
• N@10 — range: percent
  • Normalized Discounted Cumulative Gain @K computes the discounted gain of the ground-truth item at its predicted rank, normalized by the ideal DCG @K. Averaged over all users.

Input / output format

Input: Per user: a sequence of previously interacted items represented by their text descriptions and high-resolution images. No item IDs are provided.

Output: A ranked list of candidate items (top-10) from the full item pool, or item relevance scores used to compute the top-10 ranking.

Scoring recipe

def compute_hr_ndcg(ranked_list, ground_truth, k=10):
    top_k = ranked_list[:k]
    hit = 1.0 if ground_truth in top_k else 0.0
    dcg = 1.0 / math.log2(2) if ground_truth in top_k else 0.0
    idcg = 1.0 / math.log2(2)
    ndcg = dcg / idcg
    return hit, ndcg

Common pitfalls

• Ranking must be performed against the entire item pool, not a sampled set of 100 random items.
• End-to-end (E2E) training of modality encoders is prone to model collapse and requires extensive hyperparameter tuning due to 100x higher compute costs.
• Two-stage (TS) baselines freeze modality encoder parameters, while E2E optimizes them; results are not directly comparable without noting the training regime.

Evidence (verbatim from paper)

The popular H@10 (Hit Ratio @10) and N@10 (Normalized Discounted Cumulative Gain @10) are used as the evaluation metrics. Note we rank the predicted item among all items in the pool instead of drawing 100 random items.

Citation

@misc{zhang2023ninerrec,
  title={NineRec: A Benchmark Dataset Suite for Evaluating Transferable Recommendation},
  author={Zhang et al. (2023)},
  year={2023},
  note={arXiv:2309.07705}
}

• arXiv: 2309.07705