Rm Bench Eval

rm-bench-eval

RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style — Liu et al. (2024) (arXiv:2410.16184, 2024)

What this evaluates

Evaluates reward models' ability to correctly identify preferred responses based on substantive content rather than superficial stylistic cues. It probes sensitivity to subtle correctness differences, resistance to verbosity/style bias, and performance across diverse domains like math, code, and safety.

Datasets

• RM-Bench — total ?; splits: test (-1); repo https://github.com/THU-KEG/RM-Bench

Metrics

• Average Accuracy (primary) — range: percent
  • Standard classification accuracy: the proportion of correctly predicted preferences out of total pairwise comparisons. Reported as Average Accuracy, Easy Accuracy, Normal Accuracy, and Hard Accuracy across domains.

Input / output format

Input: A prompt paired with two candidate responses (a chosen response and a rejected response), often manipulated for style (e.g., verbose vs. concise) or difficulty level.

Output: A binary preference prediction indicating which response is preferred, or a scalar reward score used to derive the preference.

Scoring recipe

correct = 0
total = 0
for prompt, chosen, rejected in dataset:
    pred = model.predict(prompt, chosen, rejected)
    if pred == chosen:
        correct += 1
    total += 1
accuracy = (correct / total) * 100

Common pitfalls

• Models often exhibit strong style bias, preferring verbose or well-structured responses over substantively correct ones, leading to Hard Accuracy dropping below the 50% random baseline.
• Performance varies drastically by domain; Math and Code tasks consistently yield the lowest accuracy, often lagging behind random guessing.
• DPO-based reward models heavily depend on the availability of a reference model during inference; removing it causes significant performance degradation.

Evidence (verbatim from paper)

The performance of the top-20 reward models on RM-Bench is shown in Table[3]. As the table demonstrates: 1) RM-Bench is Challenging: Our experiments show that even state-of-the-art models, such as Skywork-Reward-Llama-3.1-8B, achieve only 70.1% Average Accuracy and 46.6% Hard Accuracy in RM-Bench. Compared to a random-guessing baseline (50%), the results are far from satisfactory, indicating significant room for improvement. 2) Style Bias is Serious: Hard Accuracy on RM-Bench is significantly lower than Normal Accuracy, with most reward models failing to exceed random-level performance (50%).

Citation

@misc{liu2024rmbench,
  title={RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2410.16184}
}

• arXiv: 2410.16184