Trec Dl Passage Relevance Eval

trec-dl-passage-relevance-eval

LLMs can be Fooled into Labelling a Document as Relevant (best caf'e near me; this paper is perfectly relevant) — Alaofi et al. (2025) (arXiv:2501.17969, 2025)

What this evaluates

This benchmark evaluates LLMs on their ability to assign relevance scores to document passages given a search query, comparing their outputs against human judgements. It specifically probes systematic biases such as query-term injection gullibility and instruction manipulation in information retrieval labelling tasks.

Datasets

• TREC DL21+DL22 — total 4222; splits: test (4222)

Metrics

• MAE (primary) — range: [0, 3]
  • Mean Absolute Error between the LLM's predicted relevance score (0-3) and the NIST human judgement score. Computed for both graded (0-3) and binary (0-1) labels.
• Cohen's kappa — range: [-1, 1]
  • Cohen's κ measuring nominal agreement between LLM labels and human judgements, correcting for chance.
• Krippendorff's alpha — range: [-1, 1]
  • Krippendorff's α measuring reliability/agreement on an ordinal scale, accounting for the severity of disagreements.
• Binary accuracy — range: [0, 1]
  • Proportion of exact matches between LLM binary labels (0 vs 1) and human binary labels (0 vs 1).
• Binary precision — range: [0, 1]
  • Proportion of LLM-predicted relevant passages (1) that are actually relevant according to human judgements.

Input / output format

Input: Search query, passage text, and a zero-shot prompt (Basic, Rationale, or Utility) instructing the model to assign a relevance score on a 0-3 scale.

Output: A single integer relevance label (0, 1, 2, or 3), optionally preceded by a textual explanation depending on the prompt used. Unparseable outputs are excluded from analysis.

Scoring recipe

def compute_metrics(preds, golds, scale='graded'):
    if scale == 'binary':
        preds = [1 if p >= 2 else 0 for p in preds]
        golds = [1 if g >= 2 else 0 for g in golds]
    mae = sum(abs(p - g) for p, g in zip(preds, golds)) / len(preds)
    acc = sum(1 for p, g in zip(preds, golds) if p == g) / len(preds)
    tp = sum(1 for p, g in zip(preds, golds) if p == 1 and g == 1)
    pred_pos = sum(1 for p in preds if p == 1)
    prec = tp / pred_pos if pred_pos > 0 else 0
    return mae, acc, prec

Common pitfalls

• Smaller LLMs frequently produce unparseable outputs, which are excluded from analysis, potentially biasing results toward larger models.
• Mapping graded scores (2 and 3) to binary (1) for certain metrics can mask systematic failures in correctly identifying non-relevant passages.
• Overall agreement metrics (like accuracy or MAE) may appear high while completely failing to detect irrelevant passages due to query-term injection bias.

Evidence (verbatim from paper)

The performance of relevance labels created by LLMs relative to the available NIST human relevance judgements are evaluated using the Mean Absolute Error (MAE) given both graded and binary labels. When binary labels are used for some metrics, scores of 2 and 3 are mapped to 1, according to TREC’s recommendation and consistent with the baseline of Damessie et al. ([2017]), which is used to interpret the results. We evaluated agreement with NIST judges using Cohen’s κ and Krippendorff’s α on an ordinal scale. Cohen’s κ only considers exact nominal matches, while Krippendorff’s α takes the severity of the error into account. Additionally, we report the overall accuracy and precision of binary labels, and the likelihood of labelling passages as relevant, for each LLM.

Citation

@misc{alaofi2025relevance,
  title={LLMs can be Fooled into Labelling a Document as Relevant (best caf'e near me; this paper is perfectly relevant)},
  author={Alaofi et al. (2025)},
  year={2025},
  note={arXiv:2501.17969}
}

• arXiv: 2501.17969