Towervision Multilingual Vl Eval

towervision-multilingual-vl-eval

TowerVision: Understanding and Improving Multilinguality in Vision-Language Models — Viveiros et al. (2025) (arXiv:2510.21849, 2025)

What this evaluates

This evaluation probes the multilingual vision-language capabilities of models across text recognition, cultural understanding, multimodal translation, and video reasoning. It specifically tests cross-lingual generalization and cultural grounding in both image and video domains across high- and low-resource languages.

Datasets

• ALM-Bench — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• cc-OCR — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• CoMMuTE — total ?; splits: test (-1)
• Multi30K — total ?; splits: test (-1)
• ViMUL-Bench — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Percentage of correct predictions on closed-form or multiple-choice questions. Computed as (correct predictions / total instances) * 100.
• xComet — range: [0, 1]
  • Cross-lingual quality metric used for Multi30K translation evaluation. Scores translation quality on a normalized scale.
• contrastive pairwise accuracy — range: [0, 1]
  • Measures whether the correct translation scores higher than incorrect alternatives in CoMMuTE.
• GPT-4o judge score — range: [0, 1]
  • Automated LLM-as-judge scoring for open-ended ViMUL-Bench responses, averaged with multiple-choice accuracy.

Input / output format

Input: Image or video frames paired with text prompts/questions in various languages (English and 20+ other languages).

Output: Text predictions (answers, translations, or captions). Open-ended responses are evaluated via LLM-as-judge.

Scoring recipe

def compute_metric(predictions, gold, task):
    if task in ['ALM-Bench', 'TextVQA', 'OCRBench', 'cc-OCR', 'ViMUL-Bench-MC']:
        return 1.0 if predictions == gold else 0.0
    elif task == 'Multi30K':
        return xcomet_score(predictions, gold)
    elif task == 'CoMMuTE':
        return contrastive_pairwise_accuracy(predictions, gold)
    elif task == 'ViMUL-Bench-Open':
        return gpt4o_judge_score(predictions, gold, prompt=shafique2025_prompt)
    return average(scores)

Common pitfalls

• Assuming all benchmarks use closed-form evaluation; ViMUL-Bench open-ended responses require LLM-as-judge scoring.
• Overlooking language-specific splits; benchmarks like ALM-Bench and cc-OCR have distinct English vs. multilingual subsets that must be evaluated separately.
• Confusing xComet with standard BLEU/chrF for Multi30K; the paper explicitly uses xComet for cross-lingual quality.

Evidence (verbatim from paper)

We report xComet (guerreiro2024xcomet) for Multi30K and contrastive pairwise accuracy for CoMMuTE.

Citation

@misc{viveiros2025towervision,
  title={TowerVision: Understanding and Improving Multilinguality in Vision-Language Models},
  author={Viveiros et al. (2025)},
  year={2025},
  note={arXiv:2510.21849}
}

• arXiv: 2510.21849