Tenrec Eval

tenrec-eval

Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems — Yuan et al. (2022) (NeurIPS 2022, 2022)

What this evaluates

Evaluates recommender systems across multiple tasks including click-through rate (CTR) prediction, sequential recommendation, and top-N item ranking. It probes cross-domain generalization, cold-start handling, and the sensitivity of ranking metrics to negative sampling strategies.

Datasets

• Tenrec — total 5000000; splits: train (-1), val (-1), test (-1); repo https://github.com/yuangh-x/2022-NIPS-Tenrec

Metrics

• AUC (primary) — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve. It measures the probability that a randomly chosen positive instance is ranked higher than a randomly chosen negative instance.
• Logloss — range: other
  • Binary cross-entropy loss: -1/N * sum(y*log(p) + (1-y)*log(1-p)). Measures the performance of a probabilistic classifier.
• NDCG@20 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 20. Compares the predicted ranking to the ground truth, applying a logarithmic discount to positions.
• HR@20 — range: [0, 1]
  • Hit Rate at rank 20. Proportion of test items that appear in the top-20 recommended items.
• Recall@20 — range: [0, 1]
  • Proportion of relevant items in the ground truth that are retrieved in the top-20 recommendations.

Input / output format

Input: User and item features, interaction history (sequence of items), and contextual metadata. For CTR: user/item features and context. For Rec: sequence of previously interacted items.

Output: CTR: probability of click. Rec: ranked list of candidate items.

Scoring recipe

def compute_auc(y_true, y_pred):
    pos = y_pred[y_true == 1]
    neg = y_pred[y_true == 0]
    concordant = sum(p > n for p in pos for n in neg)
    return concordant / (len(pos) * len(neg))

def compute_ndcg_at_k(y_true, y_pred, k=20):
    ranked = np.argsort(y_pred)[::-1][:k]
    dcg = sum((2**y_true[i] - 1) / np.log2(i + 2) for i in ranked)
    ideal = sorted(y_true, reverse=True)[:k]
    idcg = sum((2**i - 1) / np.log2(j + 2) for j, i in enumerate(ideal))
    return dcg / idcg if idcg > 0 else 0

Common pitfalls

• Negative sampling strategy heavily influences results; using advanced samplers (e.g., dynamic or popularity-based) can artificially inflate accuracy compared to random sampling.
• Cold-start evaluation must strictly isolate cold users and use only their held-out interactions (25% val, 25% test) to avoid data leakage from warm users.
• Standard top-N evaluation filters out users with session length < 10, which significantly changes the user population compared to unfiltered splits.

Evidence (verbatim from paper)

Table 13: Results for CTR prediction.

Citation

@misc{yuan2022tenrec,
  title={Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems},
  author={Yuan et al. (2022)},
  year={2022},
  note={NeurIPS 2022}
}

• arXiv: 2210.10629