Judgebench Eval

judgebench-eval

JudgeBench: A Benchmark for Evaluating LLM-based Judges — Tan et al. (2024) (arXiv:2410.12784, 2024)

What this evaluates

Probes the factual and logical reliability of LLM-based judges and reward models by testing their ability to distinguish between objectively correct responses and subtly flawed ones across knowledge, reasoning, math, and coding domains.

Datasets

• JudgeBench — total ?; splits: Knowledge (-1), Reasoning (-1), Math (-1), Coding (-1); repo https://github.com/ScalerLab/JudgeBench

Metrics

• accuracy (primary) — range: percent
  • Percentage of response pairs where the aggregated two-trial verdict matches the objectively correct response. Ties are aggregated with the majority vote; inconsistent verdicts or double-ties are counted as incorrect.

Input / output format

Input: A prompt template (e.g., Vanilla or Arena-Hard) containing a question and two candidate responses (Response A and Response B).

Output: A categorical verdict: 'A > B', 'A < B', or 'A = B' (tie).

Scoring recipe

correct = 0
for pair in dataset:
    v1 = judge(pair.A, pair.B)
    v2 = judge(pair.B, pair.A)
    if (v1 in ['A>B', 'A=B'] and v2 in ['A>B', 'A=B']):
        agg = 'A>B'
    elif (v1 in ['A<B', 'A=B'] and v2 in ['A<B', 'A=B']):
        agg = 'A<B'
    else:
        agg = 'incorrect'
    if (agg == 'A>B' and pair.gold == 'A') or (agg == 'A<B' and pair.gold == 'B'):
        correct += 1
return (correct / len(dataset)) * 100

Common pitfalls

• Evaluating judges on a single response order, which introduces positional bias and distorts accuracy.
• Counting ties as correct or discarding them, whereas the protocol explicitly aggregates ties with the majority vote or marks inconsistent/double-tie outcomes as incorrect.
• Assuming high accuracy implies robust reasoning, as the benchmark is specifically constructed to be near-random for current models, exposing gaps in factual/logical verification.

Evidence (verbatim from paper)

To mitigate this, we evaluate the LLM-based judge twice, swapping the order of the response pairs in the second trial. ... if both trials yield $A > B$ or one trial gives $A > B$ and the other $A = B$ , we consider the aggregate decision to be $A > B$ . Inconsistent decisions (e.g., $A > B$ in one trial, $A < B$ in the other) or ties in both trials are deemed incorrect, as they indicate the judge is either guessing or unable to reliably distinguish between responses. ... Even a strong model like GPT-4o struggles, achieving accuracy no better than random guessing when using the vanilla prompt.

Citation

@misc{tan2024judgebench,
  title={JudgeBench: A Benchmark for Evaluating LLM-based Judges},
  author={Tan et al. (2024)},
  year={2024},
  note={arXiv:2410.12784}
}

• arXiv: 2410.12784