Mlp4rec Eval

mlp4rec-eval

MLP4Rec: A Pure MLP Architecture for Sequential Recommendations — Li et al. (2022) (arXiv:2204.11510, 2022)

What this evaluates

Evaluates a model's ability to predict the next item in a user's sequential interaction history. It probes the model's capacity to capture temporal dependencies, cross-channel correlations in item embeddings, and cross-feature interactions using only MLP-based architectures.

Datasets

• MovieLens-100k — total 100000; splits: train (-1), val (-1), test (-1)
• Amazon Beauty — total 2023070; splits: train (-1), val (-1), test (-1)

Metrics

• HR@10 — range: [0, 1]
  • Binary indicator of whether the ground-truth item appears in the model's top-10 recommendations.
• NDCG@10 (primary) — range: [0, 1]
  • Measures the quality of the top-10 ranked list by weighting the ground-truth item's position with a logarithmic discount factor.
• MRR@10 — range: [0, 1]
  • Averages the reciprocal of the rank of the ground-truth item within the top-10 recommendations across all test instances.

Input / output format

Input: A zero-padded sequence of item IDs and associated features (e.g., brand, category) up to a maximum length of 50, representing a user's historical interactions.

Output: A ranked list of top-10 candidate items predicted for the next interaction.

Scoring recipe

def compute_metrics(preds, golds, k=10):
    hr, ndcg, mrr = 0.0, 0.0, 0.0
    for pred, gold in zip(preds, golds):
        top_k = pred[:k]
        if gold in top_k:
            hr += 1.0
            rank = top_k.index(gold) + 1
            mrr += 1.0 / rank
            ndcg += 1.0 / math.log2(rank + 1)
    n = len(preds)
    return hr/n, ndcg/n, mrr/n

Common pitfalls

• Negative sampling must pair exactly 100 negative items with each ground-truth item during prediction, as specified in the evaluation settings.
• Users and items with fewer than 5 interactions must be filtered out before dataset splitting to match the reported statistics.
• The maximum sequence length is fixed at 50 with zero-padding; ignoring this padding can cause misalignment during evaluation.

Evidence (verbatim from paper)

Metrics. We apply three commonly used evaluation metrics in recommender system, namely hit ratio (HR), normalized discounted cumulative gain (NDCG) and Mean Reciprocal Rank (MRR). HR measures the probability of the ground-truth item that appears in model's top-K recommendation, NDCG measures the order of the top-K recommended items generated by the recommender, and finally MRR takes the reciprocal of the ground-truth item's ranking in top-K recommendation and averages across all the evaluated items.

Citation

@misc{li2022mlp4rec,
  title={MLP4Rec: A Pure MLP Architecture for Sequential Recommendations},
  author={Li et al. (2022)},
  year={2022},
  note={arXiv:2204.11510}
}

• arXiv: 2204.11510