Vhelm Eval

vhelm-eval

VHELM: A Holistic Evaluation of Vision Language Models — Lee et al. (2024) (arXiv:2410.07112, 2024)

What this evaluates

Holistic evaluation of vision-language models across multiple dimensions including visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety.

Datasets

• VHELM Scenarios — total 915000; splits: test (-1); repo https://github.com/stanford-crfm/helm

Metrics

• scenario_score (primary) — range: other
  • Not specified in the provided section.

Input / output format

Input: Text and image inputs per scenario, processed with standard inference parameters.

Output: Model-generated text responses.

Scoring recipe

# Not specified in the provided section.
# Evaluation runs sample up to 1,000 instances per scenario.
# Total instances per run: 915K across 22 models.

Common pitfalls

• Sampling is capped at a maximum of 1,000 instances per scenario to save cost/time, which may not reflect full dataset performance.
• Standard inference parameters are maintained per model, but specific hyperparameters or prompting details are not provided in this section.

Evidence (verbatim from paper)

Our evaluation run randomly samples a maximum of 1,000 instances for each of the scenarios in order to alleviate monetary and time constraints; a single evaluation run on the 22 VLMs uses a total of 915K instances and consumes 51.6M input text tokens, 9.4M output text tokens, and 915K images.

Citation

@misc{lee2024vhelm,
  title={VHELM: A Holistic Evaluation of Vision Language Models},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2410.07112}
}

• arXiv: 2410.07112