qhjqhj00 visit-bench-eval
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use — Bitton et al. (2023) (arXiv:2308.06595, 2023)
What this evaluates
Evaluates vision-language models' ability to follow complex, real-world instructions on images. It probes open-ended generation, context-sensitive reasoning, and instruction-conditioned captioning by measuring how well model outputs align with human preferences and high-quality references.
Datasets
- VisIT-Bench — total 592; splits: test (592)
Metrics
Elo rating(primary) — range: unbounded (typically 700-1400)- Pairwise comparison metric where each human judgment is treated as a match. The difference in Elo ratings between two models estimates the win probability when pitting them against each other.
Win rate vs. reference(primary) — range: percent- Percentage of forced-choice pairwise judgments where a model's output is preferred over a human-verified GPT-4 reference output. Calculated as (number of wins against reference) / (total comparisons against reference).
Input / output format
Input: Image and a text instruction. For pairwise evaluation, the model receives the image, instruction, instruction-conditioned caption, and two candidate text responses (Response A and Response B).
Output: Text response generated by the vision-language model. For evaluation, a forced-choice preference (A or B) or a tie.
Scoring recipe
def compute_win_rate(predictions, golds):
# predictions: list of model outputs, golds: list of reference outputs
wins = sum(1 for p, g in zip(predictions, golds) if p == g)
return (wins / len(golds)) * 100
Common pitfalls
- Forced-choice setup does not allow ties, forcing annotators/LLMs to arbitrarily pick a winner when responses are equal quality.
- GPT-4 evaluator outputs 'tie' in 10-15% of cases; the authors resolve this by random choice, which may introduce noise.
- Elo ratings are highly dependent on the specific set of models evaluated and match counts, limiting cross-benchmark comparability.
Evidence (verbatim from paper)
We summarize the results with two metrics: Relative metric: Elo We follow [26] and compute Elo ratings, treating each pairwise human judgement as a "match." The difference between the Elo ratings of two different models provides an estimate for the win probability when pitting model A vs. model B. More details are in Appendix D. Absolute metric: Win rate vs. reference. We provide a win-rate vs. the human-verified reference. We use the 1.4K pairwise human judgments where one of A or B is the reference. We report the percent of cases where the human judge prefers the output from that model vs. the human-verified GPT-4 reference output.
Citation
@misc{bitton2023visitbench,
title={VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use},
author={Bitton et al. (2023)},
year={2023},
note={arXiv:2308.06595}
}
- arXiv: 2308.06595