Multimodal Rec Benchmark

multimodal-rec-benchmark

Formalizing Multimedia Recommendation through Multimodal Deep Learning — Malitesta et al. (2023) (arXiv:2309.05273, 2023)

What this evaluates

Evaluates the performance of multimodal deep learning recommender systems across five Amazon product categories. It probes both standard recommendation accuracy and beyond-accuracy dimensions such as novelty, diversity, popularity bias, and catalog coverage.

Datasets

• Amazon (Office, Toys, Beauty, Sports, Clothing) — total ?; splits: train (-1), test (-1)

Metrics

• Recall@k (primary) — range: [0, 1]
  • Fraction of relevant items retrieved in the top-k list per user, averaged over all users: (1/|U|) * sum_u(|Rel_u@k| / |Rel_u|).
• nDCG@k — range: [0, 1]
  • Normalized Discounted Cumulative Gain at k. Averages the ratio of DCG@k to ideal DCG@k over users, where DCG@k sums (2^rel-1)/log2(i+1) for ranked positions.
• EFD@k — range: [0, 1]
  • Expected Free Discovery at k. Quantifies novelty by weighting predicted relevance with the inverse collection frequency (-log2 p(i|seen, theta)) to measure long-tail item exposure.
• Gini@k — range: [0, 1]
  • Normalized Gini index at k. Measures popularity disparity in top-k lists; higher values indicate wider item diversity, lower values indicate concentration on popular items.
• APLT@k — range: [0, 1]
  • Average Percentage of Long-tail items at k. Calculates the percentage of niche (long-tail) items in the top-k recommendations, averaged over users.
• iCov@k — range: percent
  • Item coverage at k. Percentage of training catalog items that appear in any user's top-k recommendation list: |union_u(I_hat_u@k)| / |I_train| * 100%.

Input / output format

Input: User-item interaction history with timestamps, plus pre-extracted 4,096-dimensional visual embeddings and 1,024-dimensional textual embeddings for each item.

Output: A ranked list of top-k items for each user.

Scoring recipe

def compute_metrics(predictions, gold, k=20):
    recalls, ndcgs = [], []
    for u in gold:
        rel = gold[u]
        pred_k = predictions[u][:k]
        recalls.append(len(set(pred_k) & rel) / len(rel) if rel else 0)
        # nDCG follows standard DCG/IDCG ratio over ranked positions
    return {'Recall@k': sum(recalls)/len(recalls), 'nDCG@k': sum(ndcgs)/len(ndcgs)}

Common pitfalls

• Relying solely on accuracy metrics (Recall/nDCG) overlooks critical beyond-accuracy dimensions like novelty, diversity, and popularity bias.
• The validation split is created by removing 50% of the test set, which is a non-standard hold-out procedure that may affect hyperparameter selection fairness.
• Visual and textual features are pre-extracted embeddings rather than learned jointly from raw data, limiting direct comparison with end-to-end multimodal models.

Evidence (verbatim from paper)

For the recommendation accuracy, we consider the Recall@k and the nDCG@k; for the novelty and diversity, we measure the EFD@k and the Gini@k, respectively; for the popularity bias, we calculate the APTL@k; finally, as a general index of how recommendations cover the entire catalog of products, we adopt the iCov@k.

Citation

@misc{malitesta2023formalizing,
  title={Formalizing Multimedia Recommendation through Multimodal Deep Learning},
  author={Malitesta et al. (2023)},
  year={2023},
  note={arXiv:2309.05273}
}

• arXiv: 2309.05273