Bert4rec Eval

bert4rec-eval

BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer — Sun et al. (2019) (arXiv:1904.06690, 2019)

What this evaluates

Evaluates a model's ability to predict the next item in a user's sequential interaction history using bidirectional context. It probes how well the model captures long-range sequential dependencies and user preferences from implicit feedback.

Datasets

• Amazon Beauty — total ?; splits: train (-1), val (-1), test (-1)
• Steam — total ?; splits: train (-1), val (-1), test (-1)
• MovieLens 1m — total ?; splits: train (-1), val (-1), test (-1)
• MovieLens 20m — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@10 (primary) — range: [0, 1]
  • Hit Ratio at rank k: 1 if the ground truth item appears in the top-k recommended items, 0 otherwise. Averaged over all users.
• NDCG@10 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank k: log2(2) / log2(1 + rank) if the ground truth is in top-k, else 0. Normalized by ideal DCG (which is 1 for a single positive). Averaged over users.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank: 1/rank where rank is the position of the ground truth item in the predicted list. Averaged over users.

Input / output format

Input: A user's historical interaction sequence (list of item IDs) sorted by timestamp.

Output: A ranked list of candidate items (1 ground truth + 100 negative items sampled by popularity) for each user.

Scoring recipe

def compute_metrics(predictions, ground_truths):
    hr, ndcg, mrr = 0.0, 0.0, 0.0
    for pred_list, gt in zip(predictions, ground_truths):
        if gt in pred_list[:10]:
            hr += 1.0
            rank = pred_list.index(gt) + 1
            ndcg += 1.0 / math.log2(rank + 1)
            mrr += 1.0 / rank
    n = len(predictions)
    return hr/n, ndcg/n, mrr/n

Common pitfalls

• Negative items are sampled based on item popularity, not uniformly, which can inflate metrics if the model favors popular items.
• The dataset is filtered to only include users with >= 5 interactions, which changes the distribution compared to raw dumps.
• NDCG@1 is omitted in reporting because it equals HR@1 in this single-positive setting.

Evidence (verbatim from paper)

For each user, we hold out the last item of the behavior sequence as the test data, treat the item just before the last as the validation set, and utilize the remaining items for training. ... pairing each ground truth item in the test set with 100 randomly sampled negative items that the user has not interacted with. ... we employ a variety of evaluation metrics, including Hit Ratio (HR), Normalized Discounted Cumulative Gain (NDCG), and Mean Reciprocal Rank (MRR). ... we report HR and NDCG with k = 1, 5, 10.

Citation

@misc{sun2019bert4rec,
  title={BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer},
  author={Sun et al. (2019)},
  year={2019},
  note={arXiv:1904.06690}
}

• arXiv: 1904.06690