Mvl Sib Eval

mvl-sib-eval

MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching — Schmidt et al. (2025) (arXiv:2502.12852, 2025)

What this evaluates

Evaluates cross-modal and text-only topical matching capabilities of vision-language models across 205 languages. It probes whether models can correctly associate images with semantically related texts (or vice versa) in a multilingual multiple-choice setting.

Datasets

• MVL-SIB — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • Computed as the share of prompts for which the model's generated response begins with the correct ground-truth label letter (e.g., 'A', 'B'). Punctuation immediately following the letter (e.g., 'A.') is ignored.

Input / output format

Input: An image (downsampled to 640×480 pixels) and a multilingual text prompt/question, presented as a multiple-choice task with labeled options (e.g., A, B, C).

Output: A single uppercase letter corresponding to the chosen option (e.g., 'A', 'B', 'C'), optionally followed by punctuation.

Scoring recipe

def compute_metric(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        first_char = next((c for c in pred if c.isalpha()), None)
        if first_char == gold:
            correct += 1
    return correct / len(predictions)

Common pitfalls

• Only the first alphabetic character of the model's output is scored; trailing explanations or punctuation are ignored.
• Images are strictly downsampled to 640×480 pixels, which may remove fine-grained visual details required for some topical matching tasks.
• Greedy decoding (temperature=0.0) is enforced, which may disadvantage models that perform better with stochastic sampling.

Evidence (verbatim from paper)

We compute the share of prompts for which responses begin with the right letter. If the label is "A", a response such as "A." is also correct.

Citation

@misc{schmidt2025mvl_sib,
  title={MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching},
  author={Schmidt et al. (2025)},
  year={2025},
  note={arXiv:2502.12852}
}

• arXiv: 2502.12852