Amazon Stark Skb Eval

amazon-stark-skb-eval

Comparative Analysis of Neural Retriever-Reranker Pipelines for Retrieval-Augmented Generation over Knowledge Graphs in E-commerce Applications — Rumble et al. (2025) (arXiv:2602.22219, 2025)

What this evaluates

Evaluates the ability of neural retriever-reranker pipelines to accurately retrieve relevant product entities from semi-structured e-commerce knowledge graphs using natural language queries. It probes semantic matching, cross-encoder reranking effectiveness, and the impact of graph-based augmentation on retrieval precision and recall.

Datasets

• Amazon STaRK SKB — total 9100; splits: validation (910), test (9100)

Metrics

• Hit@1 (primary) — range: [0, 1]
  • Binary indicator: 1 if the first retrieved item is relevant, 0 otherwise.
• Hit@5 — range: [0, 1]
  • Binary indicator: 1 if any relevant item appears in the top 5 retrieved results, 0 otherwise.
• Recall@20 — range: [0, 1]
  • Fraction of all relevant items successfully retrieved within the top 20 results.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank: 1 divided by the rank position of the first relevant item retrieved.

Input / output format

Input: Natural language product queries.

Output: Ranked list of retrieved knowledge graph nodes/entities.

Scoring recipe

def score(retrieved, relevant):
    hit1 = 1.0 if retrieved[0] in relevant else 0.0
    hit5 = 1.0 if any(r in relevant for r in retrieved[:5]) else 0.0
    recall20 = len(set(retrieved[:20]) & set(relevant)) / max(len(relevant), 1)
    rr = 0.0
    for i, r in enumerate(retrieved):
        if r in relevant:
            rr = 1.0 / (i + 1)
            break
    return {'Hit@1': hit1, 'Hit@5': hit5, 'Recall@20': recall20, 'MRR': rr}

Common pitfalls

• Confusing the 910-query validation set with the full 9,100-query evaluation set when reporting results.
• Overlooking the massive computational cost difference between cross-encoder rerankers (e.g., 100s/query vs 0.5s/query) when comparing accuracy metrics.
• Assuming dense retrieval (FAISS) inherently outperforms sparse lexical methods (BM25) without accounting for graph augmentation strategies.

Evidence (verbatim from paper)

This analysis evaluated the performance of three information retrieval pipelines (BM25, FAISS-FLAT, and FAISS-HNSW) across 910 queries using four key metrics: HIT@1, HIT@5, RE-CALL@20, and MRR.

Citation

@misc{rumble2025amazonstark,
  title={Comparative Analysis of Neural Retriever-Reranker Pipelines for Retrieval-Augmented Generation over Knowledge Graphs in E-commerce Applications},
  author={Rumble et al. (2025)},
  year={2025},
  note={arXiv:2602.22219}
}

• arXiv: 2602.22219