Stark Eval

stark-eval

STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases — Shirley Wu et al. (2024) (arXiv:2404.13207, 2024)

What this evaluates

Evaluates retrieval models on their ability to find relevant entities in semi-structured knowledge bases using complex queries that combine textual descriptions and relational constraints. It probes joint reasoning over mixed textual-relational semantics and user-intent modeling across product, academic, and medical domains.

Datasets

• STaRK — total ?; splits: synthesized (-1), human-amazon (-1), human-mag (-1), human-prime (-1); repo https://github.com/snap-stanford/STaRK

Metrics

• Hit@k (primary) — range: [0, 1]
  • Binary indicator (1 if any ground-truth relevant item appears in the top-k predicted results, 0 otherwise). Averaged across queries. k=1 and k=5 are used.
• Recall@k — range: [0, 1]
  • Proportion of ground-truth relevant items retrieved within the top-k results. Calculated as |predicted_top_k ∩ relevant| / |relevant|. k=20 is used for synthesized queries.
• MRR — range: [0, 1]
  • Mean Reciprocal Rank. For each query, takes the reciprocal of the rank position of the first relevant item in the predicted list. Averaged across queries.

Input / output format

Input: Natural language query containing textual and relational constraints over a semi-structured knowledge base.

Output: Ranked list of candidate entities/documents returned by the retrieval model.

Scoring recipe

def compute_stark_metrics(pred_lists, gold_sets, k_hit=[1, 5], k_recall=20):
    hits, recalls, mrrs = [], [], []
    for preds, gold in zip(pred_lists, gold_sets):
        top_k_hit = set(preds[:max(k_hit)])
        top_k_rec = set(preds[:k_recall])
        hits.append(1.0 if any(g in top_k_hit for g in gold) else 0.0)
        recalls.append(len(top_k_rec & set(gold)) / len(gold) if gold else 0.0)
        mrr = 0.0
        for r, p in enumerate(preds, 1):
            if p in gold:
                mrr = 1.0 / r
                break
        mrrs.append(mrr)
    return {
        f'Hit@{k}': sum(hits) / len(hits) for k in k_hit,
        f'Recall@{k_recall}': sum(recalls) / len(recalls),
        'MRR': sum(mrrs) / len(mrrs)
    }

Common pitfalls

• Recall@k is bounded by the maximum answer length (≤20 items), so it cannot exceed 1.0 even if the model retrieves all relevant items.
• Rerankers (Claude3, GPT-4) are only evaluated on a random 10% sample of test queries due to computational cost, which may not reflect full-benchmark performance.
• Metrics are macro-averaged per query, not micro-averaged over all retrieved items, so queries with many relevant items can disproportionately influence Recall@k.

Evidence (verbatim from paper)

The performance of these models are measured using standard retrieval metrics below. • Hit@k assesses whether the correct item is among the top-k results from the model. We used k = 1 and k = 5 for evaluation. At k = 1, it evaluates the accuracy of the top recommendation; at k = 5, it examines the model's precision in a wider recommendation set. • Recall@k measures the proportion of relevant items in the top-k results. For synthesized queries, k = 20 is used, as the answer length of all of the queries in our benchmarks are equal or smaller than 20. This metric offers insight into the model’s ability to identify all relevant items, particularly in scenarios where missing any could be critical. • Mean Reciprocal Rank (MRR) is a statistic for evaluating the average effectiveness of a predictive model. It calculates the reciprocal of the rank at which the first relevant item appears in the list of predictions.

Citation

@misc{stark2024,
  title={STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases},
  author={Shirley Wu et al. (2024)},
  year={2024},
  note={arXiv:2404.13207}
}

• arXiv: 2404.13207