qhjqhj00 pairjudge-rm-eval
PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament — Liu et al. (2025) (arXiv:2501.13007, 2025)
What this evaluates
Evaluates whether a reward model can correctly select the right solution from a set of N generated candidates for mathematical reasoning problems. It probes the model's ability to perform pairwise correctness judgments and rank solutions without relying on arbitrary scalar scores.
Datasets
- MATH-500 — total ?; splits: test (-1)
- Olympiad Bench — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Pass@N accuracy is calculated as the fraction of problems where the solution selected by the Best-of-N sampling process matches the ground-truth answer. Accuracy = (number of correctly solved problems) / (total number of problems).
Input / output format
Input: A mathematical problem statement and a set of N candidate solutions (or pairs of solutions for pairwise judgment) generated by a base LLM.
Output: A binary correctness judgment (True/False) for each candidate solution, derived via chain-of-thought reasoning.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = 0
for prob in predictions:
candidates = prob['candidates']
# Select best candidate via reward model scores or knockout tournament
best = select_best(candidates)
if best['answer'] == gold['answer']:
correct += 1
return (correct / len(predictions)) * 100
Common pitfalls
- Confusing pass@N accuracy with standard pass@1; the metric measures selection success across N candidates, not single-generation accuracy.
- Assuming the reward model outputs a continuous scalar score; PairJudge RM uses a pairwise knockout tournament with binary correctness judgments via chain-of-thought.
- Failing to replicate the exact N values (16, 32, 64) used in the table, which significantly impact the reported accuracy.
Evidence (verbatim from paper)
The results are reported in terms of accuracy. The pass@1 accuracy of these three LLMs are 42.0, 73.6, and 59.2 on MATH-500, and 12.3, 35.7, and 25.9 on Olympiad Bench, respectively. @16, @32, and @64 denote the accuracy with Best-of-16, Best-of-32, and Best-of-64 sampling, respectively.
Citation
@misc{liu2025pairjudge,
title={PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament},
author={Liu et al. (2025)},
year={2025},
note={arXiv:2501.13007}
}
- arXiv: 2501.13007