Prometheus 2 Eval

prometheus-2-eval

Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models — Seungone Kim et al. (2024) (arXiv:2405.01535, 2024)

What this evaluates

This evaluation protocol assesses the capability of language models to act as automated judges for other language models. It probes two distinct paradigms: direct assessment, where a model scores a single response against a reference or rubric, and pairwise ranking, where a model selects the preferred response between two candidates. The benchmarks cover instruction-following, alignment, and fine-grained custom criteria.

Datasets

• Vicuna Bench — total 80; splits: test (80)
• MT Bench — total 80; splits: test (80)
• FLASK — total 200; splits: test (200)
• Feedback Bench — total 200; splits: test (200)
• HHH Alignment — total 221; splits: test (221)
• MT Bench Human Judgment — total 80; splits: test (80)
• Auto-J Eval — total 58; splits: test (58)
• Preference Bench — total 200; splits: test (200)

Metrics

• Pearson correlation (primary) — range: [-1, 1]
  • Measures the linear correlation coefficient between the evaluator model's predicted scores and the reference human/proprietary scores. Calculated as the covariance of the two variables divided by the product of their standard deviations.
• Spearman correlation — range: [-1, 1]
  • Measures the monotonic relationship between predicted and reference scores based on their rank order. Computed as the Pearson correlation of the rank-transformed variables.
• Kendall-Tau correlation — range: [-1, 1]
  • Measures the ordinal association between predicted and reference scores by counting concordant and discordant pairs. Defined as (concordant - discordant) / total pairs.
• Accuracy (primary) — range: [0, 1]
  • The proportion of correct predictions where the evaluator model's ranking choice matches the human judgment. Calculated as the number of matching predictions divided by the total number of evaluated pairs.

Input / output format

Input: Direct assessment: A prompt, a candidate response, and an appended reference answer or evaluation rubric. Pairwise ranking: A prompt and two candidate responses (Response A and Response B) provided without a reference answer.

Output: Direct assessment: A numerical score reflecting response quality. Pairwise ranking: A categorical label indicating 'win', 'lose', or 'tie' for the preferred response.

Scoring recipe

def evaluate_direct_assessment(pred_scores, ref_scores):
    pearson = pearsonr(pred_scores, ref_scores)
    spearman = spearmanr(pred_scores, ref_scores)
    kendall = kendalltau(pred_scores, ref_scores)
    return pearson, spearman, kendall

def evaluate_pairwise_ranking(pred_labels, ref_labels):
    valid_mask = [r != 'tie' for r in ref_labels]
    pred = [p for p, m in zip(pred_labels, valid_mask) if m]
    ref = [r for r, m in zip(ref_labels, valid_mask) if m]
    correct = sum(p == r for p, r in zip(pred, ref))
    return correct / len(ref) if ref else 0.0

Common pitfalls

• Confusing reference-based direct assessment with reference-free evaluation; the primary protocol explicitly appends the reference answer as input.
• Handling 'tie' responses inconsistently; the paper evaluates both by excluding ties ('w/o tie') and by grouping ties into direct assessment scores ('w/ tie').
• Assuming uniform scoring scales across benchmarks; FLASK and Feedback Bench use custom rubrics, while others rely on general helpfulness/harmlessness criteria.

Evidence (verbatim from paper)

In direct assessment, we conduct reference-based evaluations by appending the reference answer as the input. We use Pearson, Spearman, and Kendall-Tau as performance metrics to measure scoring correlations against reference evaluators. Moreover, we include the results of the reference-free direct assessment evaluation in Appendix[F]. In pairwise ranking, we conduct reference-free evaluations. Based on judgments assigned by humans, we use accuracy as our metric to measure agreement between evaluator LMs and humans.

Citation

@misc{kim2024prometheus2,
  title={Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models},
  author={Seungone Kim et al. (2024)},
  year={2024},
  note={arXiv:2405.01535}
}

• arXiv: 2405.01535