Kuaisim Eval

kuaisim-eval

KuaiSim: A Comprehensive Simulator for Recommender Systems — Zhao et al. (2023) (arXiv:2309.12645, 2023)

What this evaluates

Evaluates recommender system policies across three temporal granularities: request-level list-wise ranking, whole-session sequential recommendation under reinforcement learning, and cross-session user retention optimization. It measures how well simulated agents balance immediate engagement rewards with long-term user retention and list diversity.

Datasets

• KuaiRand — total ?; splits: train (-1), test (-1)
• ML-1m — total ?; splits: train (-1), test (-1)

Metrics

• Average L-reward (primary) — range: other
  • Average of item-wise immediate rewards across user requests in a mini-batch.
• Coverage — range: other
  • Number of distinct items exposed in a mini-batch.
• ILD (Intra-list diversity) — range: other
  • Estimates dissimilarity between items in each recommended list based on item embeddings.
• Whole-session reward — range: other
  • Average sum of immediate rewards for each session.
• Return day — range: other
  • Average time gap (in days) between the last request of one session and the first request of the next.
• User retention — range: percent
  • Average ratio of users visiting the system again after a session.

Input / output format

Input: User interaction history or current session context, along with a candidate item pool.

Output: A ranked list of recommended items or a sequence of items for the session.

Scoring recipe

def evaluate(predictions, gold, embeddings, user_history):
    l_rewards = [sum(item_reward(i) for i in pred) for pred in predictions]
    avg_l = sum(l_rewards) / len(l_rewards)
    max_l = max(l_rewards)
    coverage = len(set(i for pred in predictions for i in pred))
    ild = diversity_score(predictions, embeddings)
    session_rewards = [sum(r for r in sess) for sess in sessions]
    whole_sess = sum(session_rewards) / len(session_rewards)
    depth = max(len(sess) for sess in sessions)
    return_day = avg_gap_days(user_history)
    retention = returning_users / total_users
    return avg_l, max_l, coverage, ild, whole_sess, depth, return_day, retention

Common pitfalls

• Metrics are defined at different granularities (request-level vs session-level vs cross-session), making direct comparison across tasks difficult.
• Reward signals are generated by pretrained simulation models rather than real user clicks, which may introduce distributional shift.
• ILD relies on item embeddings that may not align with human-perceived diversity or business objectives.

Evidence (verbatim from paper)

List-wise reward (L-reward) is the average of item-wise immediate reward. We use both the average L-reward and the max L-reward across user requests in a mini-batch. Coverage describes the number of distinct items exposed in a mini-batch. Intra-list diversity (ILD) estimates the dissimilarity between items in each recommended list, based on item embedding.

Citation

@misc{zhao2023kuaisim,
  title={KuaiSim: A Comprehensive Simulator for Recommender Systems},
  author={Zhao et al. (2023)},
  year={2023},
  note={arXiv:2309.12645}
}

• arXiv: 2309.12645