Globo Session Rec Eval

globo-session-rec-eval

News Session-Based Recommendations using Deep Neural Networks — Moreira et al. (2018) (arXiv:1808.00076, 2018)

What this evaluates

This benchmark evaluates session-based news recommendation systems by predicting the next article a user will click based on their recent interaction history. It probes a model's ability to capture short-term user intent, handle temporal dynamics, and leverage both content and contextual features in a streaming environment.

Datasets

• Globo.com — total ?; splits: train (-1), test (-1); repo https://github.com/gabrielspmoreira/chameleon_recsys

Metrics

• HR@5 (primary) — range: [0, 1]
  • 1 if the ground-truth next item is in the top-5 predicted items, else 0. Averaged over all sessions.
• MRR@5 — range: [0, 1]
  • 1/r, where r is the rank of the ground-truth next item in the top-5 list (or 0 if not in top-5). Averaged over all sessions.

Input / output format

Input: A session represented as a sequence of clicked article IDs (padded to batch max length), augmented with per-article features (content embedding, recent popularity, recency) and user context (platform, device type).

Output: A ranked list of candidate articles (ground-truth next item plus sampled negatives) for next-click prediction.

Scoring recipe

def score(predictions, gold):
    hr5 = 1.0 if gold in predictions[:5] else 0.0
    mrr5 = 0.0
    if gold in predictions[:5]:
        rank = predictions.index(gold) + 1
        mrr5 = 1.0 / rank
    return {'HR@5': hr5, 'MRR@5': mrr5}

Common pitfalls

• Temporal leakage: Random train/test splits ignore the time-sensitive nature of news; this benchmark strictly enforces hour-by-hour temporal splits to prevent future data from leaking into training.
• Fresh item handling: Models like GRU4Rec cannot recommend unseen items, so evaluations must explicitly ignore clicks on articles published after the last training window (~2% of clicks), which can artificially inflate baseline scores.
• Negative sampling bias: The evaluation uses a specific negative sampling strategy (items from the same batch or recent global buffer); changing this strategy significantly alters the difficulty and reported metrics.

Evidence (verbatim from paper)

The Top-N evaluation metrics used in this study were Hit Rate (HR@5) (Ludewig and Jannach, 2018) which checks whether the clicked item is present in the top-5 ranked items, and Mean Reciprocal Rank (MRR@5) (Hidasi et al., 2016a) (Jugovac et al., 2018) (Ludewig and Jannach, 2018), a ranking metric, sensitive to the position of clicked item, which assigns higher score at top ranks.

Citation

@misc{moreira2018news,
  title={News Session-Based Recommendations using Deep Neural Networks},
  author={Moreira et al. (2018)},
  year={2018},
  note={arXiv:1808.00076}
}

• arXiv: 1808.00076