Multilingual Vlm Bench Eval

multilingual-vlm-bench-eval

Multilingual Training and Evaluation Resources for Vision-Language Models — Baiamonte et al. (2026) (arXiv:2604.18347, 2026)

What this evaluates

Evaluates vision-language models on translated benchmarks to measure cross-lingual transfer and check for performance degradation on English. It probes the model's ability to understand images and answer multiple-choice or yes/no questions in multiple European languages (DE, ES, FR, IT) while maintaining English proficiency.

Datasets

• MMBench (translated) — total ?; splits: test (-1)
• ScienceQA (translated) — total ?; splits: test (-1)
• MME (translated) — total ?; splits: test (-1)
• POPE (translated) — total ?; splits: test (-1)
• AI2D (translated) — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Proportion of correctly predicted options out of total instances.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall on binary yes/no answers.
• MME composite score — range: other
  • Aggregated score across multiple sub-tasks as defined by the original MME benchmark.

Input / output format

Input: Image paired with a multilingual (or English) multiple-choice or yes/no question.

Output: Model generates a single token or short string corresponding to the correct option (e.g., 'A', 'B', 'C', 'D') or 'yes'/'no'.

Scoring recipe

def compute_metrics(predictions, golds, metric_type):
    if metric_type == 'accuracy':
        return sum(p == g for p, g in zip(predictions, golds)) / len(golds)
    elif metric_type == 'F1':
        tp = sum(p == g == 'yes' for p, g in zip(predictions, golds))
        fp = sum(p == 'yes' and g == 'no' for p, g in zip(predictions, golds))
        fn = sum(p == 'no' and g == 'yes' for p, g in zip(predictions, golds))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif metric_type == 'MME_composite':
        return aggregate_mme_subtask_scores(predictions, golds)

Common pitfalls

• Using LLM-as-a-judge for answer extraction (paper explicitly mandates heuristic extraction).
• Forgetting to average results across the four non-English languages (DE, ES, FR, IT) when reporting multilingual performance.
• Misinterpreting the MME composite score, which aggregates across multiple sub-tasks rather than being a simple accuracy.

Evidence (verbatim from paper)

For AI2D, ScienceQA and MMBench, we report the accuracy, while for POPE we report F1 score on binary yes/no answers, while for MME, we used the original composite score. We perform the evaluation with VLMEvalKit using heuristic answer extraction (no LLM-as-a-judge) and greedy decoding.

Citation

@misc{baiamonte2026multipixmo,
  title={Multilingual Training and Evaluation Resources for Vision-Language Models},
  author={Baiamonte et al. (2026)},
  year={2026},
  note={arXiv:2604.18347}
}

• arXiv: 2604.18347