# Mvl Sib Eval

> Evaluates cross-modal and text-only topical matching capabilities of vision-language models across 205 languages. It probes whether models can correctly associate images with semantically related texts (or vice versa) in a multilingual multiple-choice setting. Use when the user wants to benchmark on MVL-SIB, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/mvl-sib-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/mvl-sib-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/mvl-sib-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/mvl-sib-eval

---


# mvl-sib-eval

> MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching — Schmidt et al. (2025) (arXiv:2502.12852, 2025)

## What this evaluates

Evaluates cross-modal and text-only topical matching capabilities of vision-language models across 205 languages. It probes whether models can correctly associate images with semantically related texts (or vice versa) in a multilingual multiple-choice setting.

## Datasets

- **MVL-SIB** — total ?; splits: test (-1)

## Metrics

- `accuracy` **(primary)** — range: [0, 1]
  - Computed as the share of prompts for which the model's generated response begins with the correct ground-truth label letter (e.g., 'A', 'B'). Punctuation immediately following the letter (e.g., 'A.') is ignored.

## Input / output format

**Input**: An image (downsampled to 640×480 pixels) and a multilingual text prompt/question, presented as a multiple-choice task with labeled options (e.g., A, B, C).

**Output**: A single uppercase letter corresponding to the chosen option (e.g., 'A', 'B', 'C'), optionally followed by punctuation.

## Scoring recipe

```python
def compute_metric(predictions, gold_labels):
    correct = 0
    for pred, gold in zip(predictions, gold_labels):
        first_char = next((c for c in pred if c.isalpha()), None)
        if first_char == gold:
            correct += 1
    return correct / len(predictions)
```

## Common pitfalls

- Only the first alphabetic character of the model's output is scored; trailing explanations or punctuation are ignored.
- Images are strictly downsampled to 640×480 pixels, which may remove fine-grained visual details required for some topical matching tasks.
- Greedy decoding (temperature=0.0) is enforced, which may disadvantage models that perform better with stochastic sampling.

## Evidence (verbatim from paper)

> We compute the share of prompts for which responses begin with the right letter. If the label is "A", a response such as "A." is also correct.

## Citation

```bibtex
@misc{schmidt2025mvl_sib,
  title={MVL-SIB: A Massively Multilingual Vision-Language Benchmark for Cross-Modal Topical Matching},
  author={Schmidt et al. (2025)},
  year={2025},
  note={arXiv:2502.12852}
}
```

- arXiv: 2502.12852