E Commerce Related Search Eval

e-commerce-related-search-eval

AI Guided Accelerator For Search Experience — Yetukuri et al. (2025) (arXiv:2508.05649, 2025)

What this evaluates

Evaluates the effectiveness of AI-generated related search queries in an e-commerce setting by measuring their ability to drive user engagement and purchases compared to a production baseline.

Datasets

• eBay user interaction logs — total ?; splits: test (-1)

Metrics

• click-through rate (CTR) (primary) — range: percent
  • Standard industry metric: total clicks divided by total impressions. Reported as relative percentage change against a production baseline.
• conversion rate — range: percent
  • Standard industry metric: total conversions (purchases) divided by total clicks. Reported as relative percentage change against a production baseline.

Input / output format

Input: Anonymized user interaction logs (impressions, clicks, purchases) over a two-week lookback window, along with mined query sequences and LLM-generated candidate queries.

Output: A ranked list of related search query suggestions presented to users.

Scoring recipe

def compute_metrics(impressions, clicks, conversions):
    ctr = sum(clicks) / sum(impressions)
    conv_rate = sum(conversions) / sum(clicks)
    return ctr, conv_rate

# Relative improvement vs baseline
rel_ctr = (ctr_new - ctr_baseline) / ctr_baseline
rel_conv = (conv_rate_new - conv_rate_baseline) / conv_rate_baseline

Common pitfalls

• Metrics are reported exclusively as relative percentage changes, obscuring absolute performance levels.
• Evaluation uses proprietary, anonymized eBay logs with no public access, size, or train/val/test splits.
• Baseline is a live production system rather than a standardized academic model, hindering direct cross-paper comparison.

Evidence (verbatim from paper)

We evaluate the effectiveness of our proposed approach in the context of the Related Search (RS) task, using click-through rate (CTR) and conversion rate as key proxies for assessing recommendation quality. Our evaluation is structured around two stages in the pipeline: (i) output from the system up to the Intent Filtering stage (as depicted in Figure[1]), and (ii) output from the LLM Alternator, which generates additional converging queries through in-context learning. To benchmark performance, we conduct a comparative analysis against the production RS system deployed at eBay, using a two-week lookback window of anonymized user interaction data.

Citation

@misc{yetukuri2025aiguided,
  title={AI Guided Accelerator For Search Experience},
  author={Yetukuri et al. (2025)},
  year={2025},
  note={arXiv:2508.05649}
}

• arXiv: 2508.05649