Blair Retrieval Recommendation Eval

blair-retrieval-recommendation-eval

Bridging Language and Items for Retrieval and Recommendation — Hou et al. (2024) (arXiv:2403.03952, 2024)

What this evaluates

Evaluates a model's ability to align natural language reviews with item metadata for downstream recommendation and search tasks. It probes sequential next-item prediction, conventional keyword-based product retrieval, and complex long-context product search.

Datasets

• Amazon REVIEWS 2023 (Beauty, Games, Baby) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/hyp1231/AmazonReviews2023
• ESCI — total ?; splits: test (27643)
• Amazon-C4 — total ?; splits: test (21223)

Metrics

• NDCG@10 (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 10. Computes the weighted sum of relevance scores (binary 1 for relevant, 0 otherwise) discounted by log2(rank+1), normalized by the ideal DCG for the top 10 results.
• NDCG@100 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at rank 100. Same as NDCG@10 but evaluated over the top 100 retrieved items.

Input / output format

Input: Sequential recommendation: chronological sequence of historical item IDs paired with their metadata (title, features, description). Product search: user query (short keyword phrase or long complex context) and a candidate pool of item metadata.

Output: Ranked list of candidate items (top-K) based on computed relevance scores.

Scoring recipe

def ndcg_at_k(relevant_items, predicted_ranking, k):
    dcg = 0.0
    for i, item in enumerate(predicted_ranking[:k]):
        if item in relevant_items:
            dcg += 1.0 / math.log2(i + 2)
    idcg = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_items), k)))
    return dcg / idcg if idcg > 0 else 0.0

Common pitfalls

• Splits are created by absolute timestamps (8:1:1 ratio), not random or chronological tail-sampling.
• The authors explicitly avoid 5-core filtering to prevent distribution shift toward popular items.
• For product search, candidate pools are constructed by randomly sampling 50 in-domain items per query, not using the full catalog.

Evidence (verbatim from paper)

We evaluate the models on the test set using the model that achieves the best ranking performance (NDCG@10) on the validation set. Table 7: Performance comparison of different methods on conventional product search (ESCI) and complex product search (Amazon-C4) tasks. We report the NDCG@100 metric.

Citation

@misc{hou2024blair,
  title={Bridging Language and Items for Retrieval and Recommendation},
  author={Hou et al. (2024)},
  year={2024},
  note={arXiv:2403.03952}
}

• arXiv: 2403.03952