Prometheus Vision Eval

prometheus-vision-eval

Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation — Lee et al. (2024) (arXiv:2401.06591, 2024)

What this evaluates

Evaluates the fine-grained judgment capability of vision-language models by scoring generated text outputs against instance-specific rubrics and reference answers. It measures alignment with human preferences and state-of-the-art VLM judges across instruction following, VQA, and captioning tasks.

Datasets

• LLaVA-Bench — total 60; splits: test (60)
• VisIT-Bench — total 500; splits: test (500)
• Perception-Bench — total 500; splits: test (500)
• OKVQA — total 500; splits: test (500)
• VQAv2 — total 500; splits: test (500)
• TextVQA — total 500; splits: test (500)
• COCO-Captions — total 500; splits: test (500)
• NoCaps — total 500; splits: test (500)

Metrics

• Pearson correlation (primary) — range: [-1, 1]
  • Measures the linear correlation between the evaluator model's scores and the reference scores (human or GPT-4V). Calculated as the covariance of the two variables divided by the product of their standard deviations.
• Kendall-Tau correlation — range: [-1, 1]
  • Measures the ordinal association between two ranked lists of scores. Counts concordant and discordant pairs to assess ranking agreement.
• Spearman correlation — range: [-1, 1]
  • Measures the monotonic relationship between two ranked score lists. Computed as the Pearson correlation between the rank variables.
• Pairwise Preference Win-rate — range: [0, 1]
  • The percentage of instances where the evaluator model's generated feedback is preferred over a baseline model's feedback in direct human pairwise comparisons.

Input / output format

Input: Image, text instruction or question, reference answer (for VQA/captioning tasks), and a fine-grained, instance-specific score rubric.

Output: A numerical score decision and accompanying language feedback explaining the score.

Scoring recipe

def compute_metrics(model_scores, ref_scores, pairwise_feedbacks):
    pearson = pearsonr(model_scores, ref_scores)
    kendall = kendalltau(model_scores, ref_scores)
    spearman = spearmanr(model_scores, ref_scores)
    
    wins = sum(1 for m, b in pairwise_feedbacks if human_prefers(m, b))
    win_rate = wins / len(pairwise_feedbacks)
    return pearson, kendall, spearman, win_rate

Common pitfalls

• Language model baselines cannot process images directly; they require a separate captioning step via LLaVA-1.5, which may lose visual details critical for scoring.
• Rubrics are instance-specific and generated differently for human vs. GPT-4V evaluation setups, making cross-setup score comparisons invalid without normalization.
• VQA and captioning benchmarks use short ground-truth answers, while instruction-following benchmarks use long-form outputs, requiring distinct rubric generation and scoring strategies.

Evidence (verbatim from paper)

Then, we measure the correlation of the scoring decision by employing Pearson, Kendall-Tau, and Spearman as our metrics. Next, we ask human annotators to compare 2 language feedbacks that are sampled from either GPT-4, GPT-4V, or Prometheus-Vision (13B) and choose which one is better. Then, we measure the Pairwise Preference Win-rate between the 3 candidates.

Citation

@misc{lee2024prometheusvision,
  title={Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2401.06591}
}

• arXiv: 2401.06591