# Vhelm Eval

> Holistic evaluation of vision-language models across multiple dimensions including visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety. Use when the user wants to benchmark on VHELM Scenarios, or asks about evaluating this task. Reports scenario_score.

- Skill: `qhjqhj00/vhelm-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/vhelm-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/vhelm-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/vhelm-eval

---


# vhelm-eval

> VHELM: A Holistic Evaluation of Vision Language Models — Lee et al. (2024) (arXiv:2410.07112, 2024)

## What this evaluates

Holistic evaluation of vision-language models across multiple dimensions including visual perception, knowledge, reasoning, bias, fairness, multilinguality, robustness, toxicity, and safety.

## Datasets

- **VHELM Scenarios** — total 915000; splits: test (-1); repo https://github.com/stanford-crfm/helm

## Metrics

- `scenario_score` **(primary)** — range: other
  - Not specified in the provided section.

## Input / output format

**Input**: Text and image inputs per scenario, processed with standard inference parameters.

**Output**: Model-generated text responses.

## Scoring recipe

```python
# Not specified in the provided section.
# Evaluation runs sample up to 1,000 instances per scenario.
# Total instances per run: 915K across 22 models.
```

## Common pitfalls

- Sampling is capped at a maximum of 1,000 instances per scenario to save cost/time, which may not reflect full dataset performance.
- Standard inference parameters are maintained per model, but specific hyperparameters or prompting details are not provided in this section.

## Evidence (verbatim from paper)

> Our evaluation run randomly samples a maximum of 1,000 instances for each of the scenarios in order to alleviate monetary and time constraints; a single evaluation run on the 22 VLMs uses a total of 915K instances and consumes 51.6M input text tokens, 9.4M output text tokens, and 915K images.

## Citation

```bibtex
@misc{lee2024vhelm,
  title={VHELM: A Holistic Evaluation of Vision Language Models},
  author={Lee et al. (2024)},
  year={2024},
  note={arXiv:2410.07112}
}
```

- arXiv: 2410.07112