Next Basket Recommendation Eval

next-basket-recommendation-eval

A Systematical Evaluation for Next-Basket Recommendation Algorithms — Zhufeng Shao et al. (arXiv:2209.02892, 2022)

What this evaluates

Evaluates the ability of recommendation algorithms to predict the next basket of items for a user based on their historical purchase sequences. It probes sequential modeling capabilities, handling of item frequency and recency, and robustness across datasets with varying basket lengths and purchase patterns.

Datasets

• TaFeng — total ?; splits: test (-1)
• Instacart — total ?; splits: test (-1)
• Dunnhumby — total ?; splits: test (-1)

Metrics

• Recall@K (primary) — range: [0, 1]
  • Measures the proportion of ground-truth items correctly recommended in the top-K list. Formula: Recall@K = |S' ∩ S| / |S|, where S is the set of ground-truth items and S' is the predicted top-K set.
• Precision@K — range: [0, 1]
  • Measures the proportion of recommended items that are in the ground-truth set. Formula: Precision@K = |S' ∩ S| / K.
• F1-Score@K — range: [0, 1]
  • Harmonic mean of Precision@K and Recall@K. Formula: F1@K = 2 * Precision@K * Recall@K / (Precision@K + Recall@K).
• PHR@K — range: [0, 1]
  • Person-wise Hit Ratio: fraction of users whose ground-truth items appear in the recommendation list. Formula: PHR@K = (1/N) * Σ hr(i), where hr(i)=1 if intersection > 0 else 0, and N is the number of testing users.
• NDCG@K — range: [0, 1]
  • Normalized Discounted Cumulative Gain: ranking-based measure sensitive to position. Formula: NDCG@K = (1 / Σ_{j=1}^{|S|} 1/log2(j+1)) * Σ_{i=1}^K ∂(S',S)/log2(i+1), where ∂ returns 1 if item is in ground-truth.
• MAP@K — range: [0, 1]
  • Mean Average Precision: mean of AP across users. AP(i) = (1/m) Σ_{i=1}^m m/p_i, where m is number of ground-truth items in top-K and p_i is their position. MAP@K = (1/N) Σ AP(i).
• MRR@K — range: [0, 1]
  • Mean Reciprocal Rank: mean of inverse rank of the first correctly recommended item. Formula: MRR@K = (1/N) Σ 1/q_i, where q_i is the rank of the first hit in the top-K list.

Input / output format

Input: Historical basket sequence (ordered list of item IDs) for a specific user.

Output: A ranked recommendation list (predicted basket) of fixed size K.

Scoring recipe

def compute_metrics(preds, golds, K):
    recalls, precisions, ndcgs, mrrs, phrs, aps = [], [], [], [], [], []
    for pred, gold in zip(preds, golds):
        pred_set = set(pred[:K])
        gold_set = set(gold)
        hits = pred_set & gold_set
        recalls.append(len(hits) / len(gold_set) if gold_set else 0)
        precisions.append(len(hits) / K)
        # NDCG
        dcg = sum(1 / math.log2(i + 2) for i, item in enumerate(pred[:K]) if item in gold_set)
        idcg = sum(1 / math.log2(i + 2) for i in range(min(len(gold_set), K)))
        ndcgs.append(dcg / idcg if idcg > 0 else 0)
        # MRR
        rr = next((1 / (i + 1) for i, item in enumerate(pred[:K]) if item in gold_set), 0)
        mrrs.append(rr)
        # PHR
        phrs.append(1.0 if hits else 0.0)
        # AP
        m = len(hits)
        ap = sum((m / (i + 1)) for i, item in enumerate(pred[:K]) if item in gold_set) / m if m > 0 else 0
        aps.append(ap)
    return {
        'Recall@K': sum(recalls)/len(recalls),
        'Precision@K': sum(precisions)/len(precisions),
        'F1@K': 2*sum(precisions)/len(precisions)*sum(recalls)/len(recalls) / (sum(precisions)/len(precisions) + sum(recalls)/len(recalls)) if (sum(precisions)+sum(recalls)) else 0,
        'PHR@K': sum(phrs)/len(phrs),
        'NDCG@K': sum(ndcgs)/len(ndcgs),
        'MAP@K': sum(aps)/len(aps),
        'MRR@K': sum(mrrs)/len(mrrs)
    }

Common pitfalls

• Prior works use inconsistent datasets and evaluation protocols, making cross-paper comparison difficult.
• Deep learning models do not consistently outperform simple baselines like TOP; performance heavily depends on dataset characteristics like average basket length.
• Failing to include the TOP baseline leads to unfair comparisons, as item frequency alone is highly predictive in next-basket tasks.

Evidence (verbatim from paper)

In order to comprehensively and fairly evaluate different approaches, we adopt the following 7 metrics: Recall, Precision, F1-Score, Person-wise Hit Ratio (PHR), Normalized Discounted Cumulative Gain (NDCG), Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR)... Following the settings of traditional NBR work, for each approach, we will recommend a predicted basket (i.e., recommendation list) with a fixed size K for evaluation. All the metrics are calculated across all predicted baskets, and all metrics have the same characteristic: the larger value, the better performance.

Citation

@misc{shao2022systematical,
  title={A Systematical Evaluation for Next-Basket Recommendation Algorithms},
  author={Zhufeng Shao et al.},
  year={2022},
  note={arXiv:2209.02892}
}

• arXiv: 2209.02892