Qb4rec Eval

qb4rec-eval

Q-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation — Huang et al. (2025) (arXiv:2512.02474, 2025)

What this evaluates

Evaluates sequential recommendation models on their ability to predict the next item in a user's interaction history using multimodal item features. It probes cross-domain generalization, the effectiveness of discrete semantic tokenization, and robustness in sparse interaction scenarios.

Datasets

• Amazon Product Reviews (Instruments, Arts, Games) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• HR@K (primary) — range: [0, 1]
  • Hit Ratio at rank K. Returns 1 if the ground-truth item appears in the top-K predicted items, else 0. Averaged over all test instances.
• NDCG@K (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank K. Computes 1/log2(1+rank) for the ground-truth item, normalized by the ideal DCG (which is 1.0 for a single relevant item). Averaged over test instances.

Input / output format

Input: User interaction history sequence (item IDs) paired with multimodal item features (text descriptions, images) and structural IDs.

Output: A ranked list of candidate items for next-item prediction.

Scoring recipe

def compute_hr_ndcg(ranked_list, ground_truth, k):
    if ground_truth in ranked_list[:k]:
        rank = ranked_list.index(ground_truth) + 1
        dcg = 1.0 / math.log2(1 + rank)
        return 1.0, dcg
    return 0.0, 0.0

hr_avg = sum(compute_hr_ndcg(preds, gt, k)[0] for preds, gt in test_data) / len(test_data)
ndcg_avg = sum(compute_hr_ndcg(preds, gt, k)[1] for preds, gt in test_data) / len(test_data)

Common pitfalls

• The evaluation uses a leave-one-out split where the most recent interaction is held out for testing and the second most recent for validation, which differs from random or chronological splits.
• The model is pre-trained on six source domains before fine-tuning on the three target domains; skipping pretraining breaks the transfer learning protocol.
• Multimodal features require specific encoders (LLaMA for text, CLIP-ViT for images) that are not part of the standard recommendation pipeline and must be replicated exactly.

Evidence (verbatim from paper)

We evaluate model performance using two widely adopted ranking metrics: Hit Ratio (HR@K) and Normalized Discounted Cumulative Gain (NDCG@K) with $K = 1, 5, 10$. Following previous work[16], We use the leave-one-out for evaluation, where the most recent interaction is held out for testing and the second most recent for validation.

Citation

@misc{huang2025qb4rec,
  title={Q-BERT4Rec: Quantized Semantic-ID Representation Learning for Multimodal Recommendation},
  author={Huang et al. (2025)},
  year={2025},
  note={arXiv:2512.02474}
}

• arXiv: 2512.02474