Shopping Queries Eval

shopping-queries-eval

Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search — Reddy et al. (2022) (arXiv:2206.06588, 2022)

What this evaluates

Evaluates e-commerce search models on query-product semantic matching, ranking, and multiclass relevance classification. It probes the system's ability to distinguish between Exact matches, Substitutes, Complements, and Irrelevant products across English, Spanish, and Japanese.

Datasets

• Shopping Queries Dataset — total 2600000; splits: train (-1), val (400), test (-1)

Metrics

• nDCG (primary) — range: [0, 1]
  • Normalized Discounted Cumulative Gain at a fixed cutoff K. Measures ranking quality by comparing the discounted gain of the predicted ranking against the ideal ranking based on graded relevance labels (Exact=1.0, others=0.0 for the ranking task).
• Micro F1 — range: [0, 1]
  • Micro-averaged F1 score for multiclass classification. Computed globally by aggregating true positives, false positives, and false negatives across all classes (Exact, Substitute, Complement, Irrelevant) before calculating precision and recall.

Input / output format

Input: Query string and product title (concatenated or separately encoded depending on the model architecture).

Output: For ranking: a ranked list of product IDs/titles. For classification: a single class label from {Exact, Substitute, Complement, Irrelevant}.

Scoring recipe

def compute_metrics(preds, golds, k=10):
    # nDCG for Task 1 (ranking)
    pred_rels = [1.0 if g == 'Exact' else 0.0 for g in golds]
    ideal_rels = sorted(pred_rels, reverse=True)
    dcg = sum((2**r - 1) / log2(i + 2) for i, r in enumerate(preds[:k]))
    idcg = sum((2**r - 1) / log2(i + 2) for i, r in enumerate(ideal_rels[:k]))
    ndcg = dcg / idcg if idcg > 0 else 0.0
    
    # Micro F1 for Tasks 2 & 3 (classification)
    tp = fp = fn = 0
    for p, g in zip(preds, golds):
        if p == g: tp += 1
        else: fp += 1; fn += 1
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    micro_f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0.0
    return {'nDCG': ndcg, 'Micro F1': micro_f1}

Common pitfalls

• Using binary relevance (Exact=1, others=0) for ranking evaluation instead of leveraging the full graded ESCI labels, which loses nuance in substitute/complement discrimination.
• Comparing BM25 with default non-Japanese preprocessing against fine-tuned neural models, which unfairly penalizes the averaged metric across locales.
• Assuming model performance generalizes equally across languages without adjusting tokenization or preprocessing for Japanese/Spanish.

Evidence (verbatim from paper)

Table 4 shows the results of the baselines in the public test set for the three tasks. Results are also presented broken down by language (English, Spanish, Japanese) corresponding to the US, ES and JP locales. For Task 1, we can see that the neural approach gets a much better nDCG results than the Terrie-BM25 counterpart (0.852 vs. 0.551). ... For the other two classification tasks, the BERT-based MLP classifier obtains results that are clearly better for English than for Spanish and Japanese (e.g., for Task 2, compare the F1 for English and Spanish, 0.685 vs. 0.580).

Citation

@misc{reddy2022shopping,
  title={Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Search},
  author={Reddy et al. (2022)},
  year={2022},
  note={arXiv:2206.06588}
}

• arXiv: 2206.06588