Style4rec Eval

style4rec-eval

Style4Rec: Enhancing Transformer-based E-commerce Recommendation Systems with Style and Shopping Cart Information — Ugurlu et al. (2025) (arXiv:2501.09354, 2025)

What this evaluates

Evaluates a model's ability to perform sequential product recommendation in an e-commerce setting by predicting the next item in a user session. It specifically probes how well the model leverages historical interaction sequences, visual style embeddings, and shopping cart data to rank relevant products.

Datasets

• Style4Rec E-commerce Dataset — total 38117; splits: train (-1), val (-1), test (-1)

Metrics

• HR@5 (primary) — range: [0, 1]
  • Hit Ratio at k=5: 1 if the ground truth item appears in the top-5 predicted items, else 0.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k=5: sum of 1/log2(rank+1) for hits in top-5, normalized by the ideal DCG.
• MRR@5 — range: [0, 1]
  • Mean Reciprocal Rank at k=5: 1 divided by the rank of the first hit in the top-5 list.

Input / output format

Input: A sequence of product IDs representing a user's session (pageviews, purchases, or cart additions), truncated or padded to a maximum length of 20. Negative sampling provides 100 candidate negative items per session.

Output: A ranked list of top-k product IDs (k ∈ {5, 10, 20}) predicted as the next item in the session.

Scoring recipe

def compute_metrics(predictions, ground_truth, k=5):
    top_k = predictions[:k]
    hr = 1.0 if ground_truth in top_k else 0.0
    if hr == 0:
        return 0.0, 0.0, 0.0
    rank = predictions.index(ground_truth) + 1
    ndcg = 1.0 / math.log2(rank + 1)
    mrr = 1.0 / rank
    return hr, ndcg, mrr

Common pitfalls

• Uses a strict chronological time-based split (14/2/2 months) rather than random user/session splitting, which is critical for sequential recommendation evaluation.
• Negative sampling is fixed at 100 items per session, meaning metrics are computed over a 101-item candidate set rather than the full catalog.
• Preprocessing removes repeated final products in sessions, which differs from standard session-based recommendation benchmarks that typically keep raw sequences.

Evidence (verbatim from paper)

To assess the performance of our recommendation system, we calculate several evaluation metrics, including Hit Ratio (HR), Mean Reciprocal Rank (MRR), and Normalized Discounted Cumulative Gain (NDCG) (He et al. [2017]; He, Kang, and McAuley [2017]). These metrics are calculated for product recommendation list lengths of 5, 10, and 20, providing a comprehensive evaluation of our system’s effectiveness in generating accurate and relevant recommendations.

Citation

@misc{ugurlu2025style4rec,
  title={Style4Rec: Enhancing Transformer-based E-commerce Recommendation Systems with Style and Shopping Cart Information},
  author={Ugurlu et al. (2025)},
  year={2025},
  note={arXiv:2501.09354}
}

• arXiv: 2501.09354