Vibe Eval Eval

vibe-eval-eval

Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models — Padlewski et al. (2024) (arXiv:2405.02287, 2024)

What this evaluates

Probes multimodal reasoning and visual understanding on real-world images. Specifically designed with a 'hard' subset of prompts that are unsolvable by current frontier models to measure genuine performance gaps and contamination-free generalization.

Datasets

• Vibe-Eval — total 269; splits: hard (100), normal (169); repo https://github.com/reka-ai/reka-vibe-eval

Metrics

• Vibe-Eval Score (primary) — range: percent
  • Automated score generated by the Reka Core evaluator on a 1-5 scale. The final metric is the mean score across all prompts, reported as a percentage. For robustness, the evaluator is run three times per prompt with temperature 0.4, and the mean of these runs is taken.

Input / output format

Input: Multimodal prompts consisting of real-world images paired with text instructions/questions. Evaluated in a zero-shot chat style without exemplars.

Output: Open-ended text responses generated by the multimodal language model.

Scoring recipe

scores = []
for prompt in dataset:
    run_scores = []
    for _ in range(3):
        response = model.generate(prompt, temperature=0.0)
        score = reka_core.evaluate(prompt, response, temperature=0.4) # Returns 1-5
        run_scores.append(score)
    scores.append(mean(run_scores))
final_score = mean(scores) * 100

Common pitfalls

• The hard subset is explicitly seeded using prompts where Reka Core fails, introducing a negative bias against Core on that split.
• Assigning partial credit for hard prompts requires domain expertise and leads to high variance among human raters.
• The automatic evaluator (Reka Core) tends to be generous and compresses the score range, reducing sensitivity to minor performance differences on small subsets.

Evidence (verbatim from paper)

For automated evaluation, we run the Core evaluator three times with a temperature of 0.4 and take the mean of the scores. Table 2: Vibe-Eval score and ranking for existing multimodal language models sorted by overall score.

Citation

@misc{padlewski2024vibeeval,
  title={Vibe-Eval: A hard evaluation suite for measuring progress of multimodal language models},
  author={Padlewski et al. (2024)},
  year={2024},
  note={arXiv:2405.02287}
}

• arXiv: 2405.02287