Kualive Eval

kualive-eval

KuaiLive: A Real-time Interactive Dataset for Live Streaming Recommendation — Qu et al. (2025) (arXiv:2508.05633, 2025)

What this evaluates

Evaluates recommendation models on live streaming data by testing their ability to rank relevant live rooms or streamers (top-K) and predict click-through probabilities (CTR), while accounting for real-time temporal dynamics and dynamic candidate pools.

Datasets

• KuaiLive — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Recall@{5, 10, 20} (primary) — range: [0, 1]
  • Fraction of ground-truth positive items appearing in the top-K predicted list. Computed over 10,000 sampled negatives per instance.
• NDCG@{5, 10, 20} — range: [0, 1]
  • Normalized Discounted Cumulative Gain at K, measuring ranking quality with position-based discounting.
• AUC — range: [0, 1]
  • Area Under the Receiver Operating Characteristic Curve, measuring the probability that a randomly chosen positive instance is ranked higher than a negative one.
• LogLoss — range: [0, inf)
  • Binary cross-entropy loss evaluating the quality of predicted click probabilities.

Input / output format

Input: For top-K: user ID, item ID, and interaction history (IDs only). For CTR: user features (genre, age, follow_num), streamer features (genre, age, live_operation_tag, fans_num), and interaction history.

Output: For top-K: a ranked list of top-K item IDs. For CTR: a predicted click probability score.

Scoring recipe

def recall_at_k(preds, gold, k):
    return len(set(preds[:k]) & set(gold)) / len(gold)

def ndcg_at_k(preds, gold, k):
    dcg = sum(1.0 / log2(i + 2) for i, item in enumerate(preds[:k]) if item in gold)
    idcg = sum(1.0 / log2(i + 2) for i in range(min(len(gold), k)))
    return dcg / idcg if idcg > 0 else 0.0

def auc(y_true, y_pred):
    return roc_auc_score(y_true, y_pred)

def logloss(y_true, y_pred):
    return -mean(y_true * log(y_pred) + (1 - y_true) * log(1 - y_pred))

Common pitfalls

• Candidate items must be filtered by their active start/end timestamps to simulate real-time availability; using the full static item pool inflates performance unrealistically.
• Negative sampling is fixed at 10,000 items per instance for top-K evaluation, which differs from standard practice of sampling fewer negatives and can significantly impact Recall/NDCG values.
• Item definition (live room vs. streamer) drastically changes data sparsity and cold-start conditions, requiring separate evaluation runs.

Evidence (verbatim from paper)

For evaluating top-$K$ recommendation methods, we adopt a standard leave-one-out splitting strategy. Since most top-$K$ recommendation methods only rely on ID information, we do not incorporate any user or item features in the training and evaluation. To better simulate the real-world environment, we first identify candidate items that were active at the time of each interaction based on their start and end timestamps. Then, for each instance in the validation and test sets, we sample 10,000 negative items from this candidate pool. Benchmarked models are evaluated using Recall@{5, 10, 20} and NDCG@{5, 10, 20}.

To evaluate CTR prediction methods, we combine positive and negative samples and split the data based on interaction timestamps into training, validation, and testing sets with a ratio of 8:1:1. For user features, we include genre, age, and follow_num. For streamer features, we use genre, age, live_operation_tag, and fans_num. We adopt Area Under Curve (AUC) and LogLoss as evaluation metrics.

Citation

@misc{qu2025kualive,
  title={KuaiLive: A Real-time Interactive Dataset for Live Streaming Recommendation},
  author={Qu et al. (2025)},
  year={2025},
  note={arXiv:2508.05633}
}

• arXiv: 2508.05633