Relevance Judgment Eval

relevance-judgment-eval

Re-Rankers as Relevance Judges — Meng et al. (2026) (arXiv:2601.04455, 2026)

What this evaluates

Evaluates whether pointwise re-rankers can function as binary relevance judges by predicting whether a document is relevant to a query. It probes the capability of adapted ranking models to perform direct relevance classification and compares their performance against LLM-based judges.

Datasets

• TREC-DL — total ?; splits: test (-1)

Metrics

• binary accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted binary relevance labels (0 or 1) out of the total number of query-document pairs evaluated.
• Precision, Recall, MRR, MAP — range: [0, 1]
  • Standard information retrieval metrics computed under binary relevance judgments. Precision measures the fraction of retrieved relevant documents, Recall measures the fraction of relevant documents retrieved, MRR is the reciprocal rank of the first relevant document, and MAP averages precision across all relevant documents.

Input / output format

Input: Query $q$ and document $d$.

Output: Discrete relevance label $l \in {0, 1}$, where $l=1$ denotes relevant and $l=0$ denotes not relevant.

Scoring recipe

def compute_binary_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• Self-preference and cross-family bias can artificially inflate scores when re-rankers judge outputs from similar or identical models.
• Score thresholding introduces arbitrary cutoffs that may not align with ground truth relevance boundaries, affecting binary label generation.
• Circularity risks arise when the same model family is used for both retrieval and judging, violating independent evaluation assumptions.

Evidence (verbatim from paper)

many widely used IR evaluation metrics (e.g., Precision, Recall, MRR, and MAP) are commonly applied under binary relevance judgments

Citation

@misc{meng2026rerankers,
  title={Re-Rankers as Relevance Judges},
  author={Meng et al. (2026)},
  year={2026},
  note={arXiv:2601.04455}
}

• arXiv: 2601.04455