Probenc Eval

probenc-eval

ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks — Yan Yang et al. (2025) (arXiv:2503.06885, 2025)

What this evaluates

Evaluates multimodal foundation models on open-ended, expert-level queries across 10 professional domains, probing visual perception, domain knowledge, and long-context reasoning in single-round, multi-lingual, and multi-turn settings.

Datasets

• ProBench — total 4000; splits: test (4000)

Metrics

• ELO rating (primary) — range: other
  • Pairwise comparison score derived from win rates against a baseline model, de-biased by swapping presentation order. Higher values indicate stronger relative performance.
• Win Rate (WR) — range: percent
  • Percentage of pairwise comparisons won by a model against the baseline or other evaluated models.

Input / output format

Input: Multimodal prompts (images and text) representing open-ended expert queries from professional workflows.

Output: Open-ended text responses generated by the MLLM.

Scoring recipe

def compute_metrics(predictions, queries, judge='gpt-4o-2024-08-06', baseline='gpt-4o-2024-05-13'):
    wins = {m: 0 for m in predictions}
    for q in queries:
        for m1, m2 in pairwise(predictions):
            # Swap order twice to de-bias
            judge_out = judge.compare(m1.response, m2.response, q)
            wins[judge_out.winner] += 1
    elo = calculate_elo_debiased(wins, baseline)
    wr = {m: wins[m]/total for m in wins}
    return elo, wr

Common pitfalls

• Relies on an LLM-as-a-Judge pipeline rather than ground-truth labels, making results sensitive to judge model selection and prompt design.
• ELO ratings are relative to a fixed baseline (gpt-4o-2024-05-13); changing the baseline or judge shifts relative rankings.
• Open-ended outputs prevent exact-match scoring, requiring careful aggregation of pairwise wins to compute final metrics.

Evidence (verbatim from paper)

Our MLLM judge utilizes gpt-4o-2024-08-06 with greedy sampling for consistent and reproducible evaluation. For pairwise comparisons in Elo rating calculations, we set gpt-4o-2024-05-13 as the baseline, evaluate each model twice by swapping the presentation order for each user query, and de-bias the ELO ratings by following the methodology of (Li et al., 2024c).

Citation

@misc{yang2025probenc,
  title={ProBench: Judging Multimodal Foundation Models on Open-ended Multi-domain Expert Tasks},
  author={Yan Yang et al. (2025)},
  year={2025},
  note={arXiv:2503.06885}
}

• arXiv: 2503.06885