Umbrela Relevance Assessment Eval

umbrela-relevance-assessment-eval

A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look — Upadhyay et al. (2024) (arXiv:2411.08275, 2024)

What this evaluates

This protocol evaluates the reliability of automatically generated relevance judgments (via the UMBRELA tool) compared to human assessments across different workflow conditions. It measures how well LLM-generated qrels align with human qrels in ranking retrieval systems using standard IR metrics and rank correlation.

Datasets

• TREC 2024 RAG Track — total 301; splits: test (301)

Metrics

• Kendall's τ (primary) — range: [-1, 1]
  • Measures the rank correlation between two sets of evaluation scores (e.g., run-level nDCG@20 scores computed from UMBRELA qrels vs. manual qrels). Computed across run-level scores, averaged per-topic correlations, or all topic/run combinations.
• nDCG@20 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 20, evaluating early precision in ranked retrieval lists.
• nDCG@100 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at cutoff 100, capturing effectiveness over deeper ranked lists.
• Recall@100 — range: [0, 1]
  • Recall at cutoff 100, quantifying the proportion of relevant passages retrieved within the top 100 results.

Input / output format

Input: Query, retrieved ranked list of passages, and (for human/LLM assessment) the task of assigning a relevance grade to each passage. For the correlation analysis, the input is two sets of qrels (one from UMBRELA, one from human assessment) for the same set of topics/runs.

Output: Relevance grade per passage (not relevant, related, highly relevant, perfectly relevant). For the final evaluation step, a rank correlation coefficient (Kendall's τ) between two score distributions.

Scoring recipe

# Compute run-level scores for each metric using qrels
scores_umbrela = [compute_metric(run, qrels_umbrela) for run in runs]
scores_manual = [compute_metric(run, qrels_manual) for run in runs]
# Compute Kendall's tau between the two score vectors
tau = kendalltau(scores_umbrela, scores_manual)
# For per-topic avg: compute tau per topic, then average
# For all topic/runs: flatten all topic/run score pairs and compute tau

Common pitfalls

• Confusing 'run-level correlation' (correlating aggregate scores across runs) with 'per-topic average correlation' (averaging correlations computed per topic).
• Assuming LLM-assisted human workflows yield higher correlation with gold standards than fully manual assessments (the study finds no tangible benefit).
• Mixing disjoint and overlapping topic sets when computing correlations, which changes the statistical properties and comparability of the results.

Evidence (verbatim from paper)

At a high-level, our analyses compute rank correlations between evaluation scores induced by UMBRELA judgments (qrels) vs. judgments (qrels) derived by the other assessment processes. Following common practice in IR meta-evaluations, rank correlation is captured using Kendall’s τ. However, there are multiple ways to design a rank correlation analysis. ... We focus on three: nDCG@20, nDCG@100, and Recall@100.

Citation

@misc{upadhyay2024large,
  title={A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look},
  author={Upadhyay et al. (2024)},
  year={2024},
  note={arXiv:2411.08275}
}

• arXiv: 2411.08275