Trec Dl Hole Filling Eval

trec-dl-hole-filling-eval

LLMs Can Patch Up Missing Relevance Judgments in Evaluation — Upadhyay et al. (2024) (arXiv:2405.04727, 2024)

What this evaluates

Evaluates the ability of LLMs to accurately predict missing relevance judgments (holes) in information retrieval test collections. It measures how well synthetic incomplete judgments can be restored to match ground truth relevance labels across varying hole percentages.

Datasets

• TREC DL 2019/2020/2021 — total ?; splits: test (-1)

Metrics

• Kendall τ (primary) — range: [-1, 1]
  • Measures the rank correlation between predicted relevance labels and ground truth labels. Computed as (concordant pairs - discordant pairs) / total comparable pairs. Values range from -1 (perfect disagreement) to 1 (perfect agreement).

Input / output format

Input: Query-passage pairs with a subset of relevance labels (0–3) removed (marked as holes), along with the remaining observed labels and query IDs.

Output: Predicted relevance label (0, 1, 2, or 3) for each query-passage pair with a missing judgment.

Scoring recipe

def compute_kendall_tau(predictions, ground_truth):
    concordant = discordant = 0
    for i in range(len(predictions)):
        for j in range(i + 1, len(predictions)):
            diff_pred = predictions[i] - predictions[j]
            diff_gold = ground_truth[i] - ground_truth[j]
            if diff_pred * diff_gold > 0:
                concordant += 1
            elif diff_pred * diff_gold < 0:
                discordant += 1
    total = concordant + discordant
    return (concordant - discordant) / total if total > 0 else 0.0

Common pitfalls

• Holes are only removed from relevant passages (labels 1, 2, 3), not irrelevant ones (label 0), which may bias evaluation towards relevance prediction rather than full ranking.
• Synthetic hole percentages are applied uniformly across relevant classes, but real-world missingness may be non-random or query-dependent.
• Evaluation relies on LLM-generated predictions rather than human judgments, introducing model-specific biases and API cost constraints.

Evidence (verbatim from paper)

Using TREC DL datasets, it demonstrates that LLMs like Vicuña-7B and GPT-3.5-Turbo achieve strong Kendall τ correlations (0.87–0.92) with ground truth even when only 10% of judgments remain, enabling robust, automated evaluation without human bias from incomplete data. To test the effectiveness of our framework, we synthetically prepare incomplete judgments. We sample certain percentages of the complete judgments from each relevant (i.e. label: 1, 2, and 3) label and they are marked as holes for our synthetic incomplete judgments. The sampling percentages that we used for formulations are {10, 20, 30, 40, 50, 60, 70, 80, 90}.

Citation

@misc{upadhyay2024llmspatch,
  title={LLMs Can Patch Up Missing Relevance Judgments in Evaluation},
  author={Upadhyay et al. (2024)},
  year={2024},
  note={arXiv:2405.04727}
}

• arXiv: 2405.04727