Reward Model Benchmarking Eval

reward-model-benchmarking-eval

WildReward: Learning Reward Models from In-the-Wild Human Interactions — Hao Peng et al. (2026) (arXiv:2602.08829, 2026)

What this evaluates

Evaluates reward models on their ability to correctly rank or score LLM-generated responses across diverse domains like safety, mathematics, coding, and instruction following. It probes pairwise preference accuracy, robustness to superficial biases, cross-sample score calibration, and consistency in assigning absolute quality scores.

Datasets

• RewardBench — total ?; splits: test (-1)
• RM-Bench — total ?; splits: easy (-1), normal (-1), hard (-1)
• PPE — total ?; splits: Human (-1), Correctness (-1)
• JudgeBench — total ?; splits: test (-1)

Metrics

• binary choice accuracy (primary) — range: [0, 1]
  • Proportion of prompts where the model correctly selects the ground-truth chosen response over the rejected response.
• Best-of-N accuracy — range: [0, 1]
  • Proportion of prompts where the model correctly identifies the single best response among N candidates.
• Expected Calibration Error (ECE) — range: [0, 1]
  • Weighted mean of the absolute difference between predicted confidence and actual accuracy across binned predictions. Lower values indicate better calibration.
• ROC-AUC — range: [0, 1]
  • Area under the Receiver Operating Characteristic curve, measuring the probability that a positive instance is ranked higher than a negative one across the global distribution.

Input / output format

Input: For pairwise benchmarks: a prompt paired with two candidate responses (chosen and rejected). For pointwise calibration/consistency: conversation history, user query, model response, and a binary feedback label (positive/negative).

Output: For pairwise: a binary selection indicating the preferred response. For pointwise/calibration: a continuous score or margin between chosen and rejected responses.

Scoring recipe

# Pairwise Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)

# Best-of-N Accuracy
bon_correct = sum(1 for batch in batches if max_score(batch) == gold_best_idx)
bon_accuracy = bon_correct / len(batches)

# ECE (binned)
ece = sum(bin_size * abs(bin_acc - bin_conf) for bin_size, bin_acc, bin_conf in bins)

Common pitfalls

• PPE Correctness uses a Best-of-N setting rather than binary choice, so standard pairwise accuracy does not apply.
• Reward models trained with Bradley-Terry objectives often lack global score calibration; raw margins must be calibrated (e.g., via Platt scaling) before computing ECE or using scores for cross-sample filtering.
• RM-Bench splits (easy/normal/hard) test different robustness properties; reporting only the average masks performance on length/style bias or factual correctness.

Evidence (verbatim from paper)

We adopt the widely used Expected Calibration Error (ECE) as the metric, where a lower value indicates better calibration. WILDREWARD achieves a remarkably low ECE of 2.76% . This implies that, on average, the discrepancy between the predicted confidence and its actual accuracy is less than 3% .

Citation

@misc{peng2026wildreward,
  title={WildReward: Learning Reward Models from In-the-Wild Human Interactions},
  author={Hao Peng et al. (2026)},
  year={2026},
  note={arXiv:2602.08829}
}

• arXiv: 2602.08829