Shpi Recommendation Eval

shpi-recommendation-eval

Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning — Mazoure et al. (2021) (arXiv:2106.00589, 2021)

What this evaluates

Evaluates offline reinforcement learning methods for session-based recommendation systems in optimizing long-term user retention versus short-term clicks. It tests the ability of algorithms to learn from fixed logging policies and generalize to online rollouts across synthetic, simulated, and real-world recommendation environments.

Datasets

• Synthetic recommendation problem — total ?; splits: (unstated)
• RecoGym — total ?; splits: (unstated)
• HIV treatment simulator — total ?; splits: (unstated)
• Private dataset X — total 120000; splits: (unstated)

Metrics

• undiscounted test performance on true environment rewards (primary) — range: other
  • Mean of true environment rewards collected over 200 online rollouts, averaged across 5 random seeds. Higher values indicate better long-term performance.

Input / output format

Input: State/context features (e.g., moving average of previous contexts, user features, action features) and available action set.

Output: Discrete action selection (recommendation) from the available action space.

Scoring recipe

rewards = []
for seed in range(5):
  for rollout in range(200):
    state = env.reset()
    total_r = 0
    while not done:
      action = policy(state)
      state, reward, done, _ = env.step(action)
      total_r += reward
    rewards.append(total_r)
return sum(rewards) / len(rewards)

Common pitfalls

• The evaluation is strictly offline-to-online: models are trained on a fixed dataset D from a logging policy and then evaluated via online rollouts, not standard offline test sets.
• Long-term reward (LTR) is distinct from short-term click rewards; optimizing for clicks does not guarantee high LTR due to policy-induced drift.
• The private dataset lacks explicit train/val/test splits; it is used as a single fixed offline dataset for training, with evaluation done via simulation.

Evidence (verbatim from paper)

All experiments report undiscounted test performance on true environment rewards over 200 rollouts and 5 random seeds.

Citation

@misc{mazoure2021improving,
  title={Improving Long-Term Metrics in Recommendation Systems using Short-Horizon Reinforcement Learning},
  author={Mazoure et al. (2021)},
  year={2021},
  note={arXiv:2106.00589}
}

• arXiv: 2106.00589