Multimodal Rec Eval

multimodal-rec-eval

Ducho meets Elliot: Large-scale Benchmarks for Multimodal Recommendation — Attimonelli et al. (2024) (arXiv:2409.15857, 2024)

What this evaluates

Benchmarks classical and multimodal recommender systems by evaluating how different visual and textual feature extractors impact recommendation performance. Probes the trade-off between extractor complexity and recommendation accuracy across diverse e-commerce domains.

Datasets

• Office Products — total ?; splits: test (-1)
• Digital Music — total ?; splits: test (-1)
• Baby — total ?; splits: test (-1)
• Toys & Games — total ?; splits: test (-1)
• Beauty — total ?; splits: test (-1)

Metrics

• Recall — range: percent
  • Fraction of relevant items found in the top-20 recommended list.
• nDCG (primary) — range: percent
  • Normalized Discounted Cumulative Gain at rank 20, measuring ranking quality by weighting relevant items by their position.
• HR — range: percent
  • Hit Rate at rank 20; equals 1 if at least one relevant item is in the top-20 list, else 0.

Input / output format

Input: User-item interaction data paired with pre-extracted visual and textual feature vectors for each item.

Output: Top-20 ranked list of items per user.

Scoring recipe

def score(preds, gold, k=20):
    top_k = preds[:k]
    hits = sum(1 for x in top_k if x in gold)
    recall = (hits / len(gold)) * 100 if gold else 0
    dcg = sum(1.0 / math.log2(i + 2) for i, x in enumerate(top_k) if x in gold)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(gold), k)))
    ndcg = (dcg / idcg) * 100 if idcg > 0 else 0
    hr = (1.0 if hits > 0 else 0.0) * 100
    return recall, ndcg, hr

Common pitfalls

• Metrics are strictly computed on top-20 lists, not standard top-10 or top-50 cutoffs.
• Feature extractors are applied as fixed offline pipelines; the benchmark does not evaluate joint end-to-end training of extractors and recommenders.
• Datasets correspond to specific Amazon review subcategories, which may limit direct comparison with full-dataset baselines.

Evidence (verbatim from paper)

For instance, it is important to mention that LATTICE achieved the highest performance across all metrics on Office Products, while FREEDOM overcame other approaches on the remaining datasets, except for the HR on Digital Music.

Citation

@misc{attimonelli2024ducho,
  title={Ducho meets Elliot: Large-scale Benchmarks for Multimodal Recommendation},
  author={Attimonelli et al. (2024)},
  year={2024},
  note={arXiv:2409.15857}
}

• arXiv: 2409.15857