Llmjudge Eval

llmjudge-eval

Judging the Judges: A Collection of LLM-Generated Relevance Judgements — Rahmani et al. (2025) (arXiv:2502.13908, 2025)

What this evaluates

This benchmark evaluates the agreement and ranking consistency of LLM-generated relevance judgments against human assessments. It probes whether automated scoring methods can reliably replicate human relevance labels and maintain correct document ordering for information retrieval tasks.

Datasets

• LLMJudge test set — total ?; splits: test (-1)

Metrics

• Cohen's \kappa (primary) — range: [-1, 1]
  • Measures inter-rater agreement between LLM and human relevance labels, correcting for chance agreement. Calculated as kappa = (Po - Pe) / (1 - Pe), where Po is observed agreement and Pe is expected agreement by chance.
• Krippendorff's \alpha — range: [-1, 1]
  • A reliability coefficient measuring agreement among multiple raters or at different levels of measurement, robust to missing data and small sample sizes.
• Kendall's \tau — range: [-1, 1]
  • A rank correlation coefficient measuring the correspondence between two ranked lists (LLM vs human). Ranges from -1 (complete disagreement) to 1 (complete agreement).

Input / output format

Input: Query-document pairs with human relevance judgments (typically on a 4-point scale) used as ground truth for evaluation.

Output: Relevance labels (4-point scale or binarized) and/or relevance scores generated by the LLM judge.

Scoring recipe

def compute_cohens_kappa(llm_labels, human_labels):
    # Align labels and remove mismatches
    labels = list(zip(llm_labels, human_labels))
    # Compute confusion matrix
    matrix = np.zeros((4, 4))
    for l, h in labels:
        matrix[l][h] += 1
    # Observed agreement
    Po = np.trace(matrix) / matrix.sum()
    # Expected agreement
    row_totals = matrix.sum(axis=1)
    col_totals = matrix.sum(axis=0)
    Pe = np.sum(row_totals * col_totals) / (matrix.sum() ** 2)
    # Kappa
    return (Po - Pe) / (1 - Pe) if (1 - Pe) > 0 else 0

Common pitfalls

• High ranking correlation (Kendall's \tau) does not guarantee high label agreement (Cohen's \kappa); models often preserve order but misclassify absolute relevance levels.
• Binarization thresholds drastically change \kappa scores; aggregating labels differently (e.g., 0-1 vs 2-3) yields inconsistent reliability estimates.
• Average assigned scores vary widely across methods despite similar ranking performance, indicating systematic calibration bias rather than ranking failure.

Evidence (verbatim from paper)

The results of the LLMJudge challenge, as presented in Table [3], reveal significant variability in performance across the evaluated metrics, including Cohen’s Kappa ($\kappa$), Krippendorff’s Alpha ($\alpha$), Kendall’s Tau ($\tau$), and Spearman’s Rho ($\rho$).

Citation

@misc{rahmani2025judging,
  title={Judging the Judges: A Collection of LLM-Generated Relevance Judgements},
  author={Rahmani et al. (2025)},
  year={2025},
  note={arXiv:2502.13908}
}

• arXiv: 2502.13908