Mm Judgebench Eval

mm-judgebench-eval

Lost in Translation: Do LVLM Judges Generalize Across Languages? — Md Tahmid Rahman Laskar et al. (arXiv:2604.19405, 2026)

What this evaluates

Evaluates the cross-lingual generalization and robustness of Large Vision-Language Models (LVLMs) acting as automated judges. It probes their ability to correctly rank paired multimodal responses across 25 languages while measuring susceptibility to positional and length biases.

Datasets

• MM-JudgeBench — total 60000; splits: test (-1); repo https://github.com/tahmedge/mm-judgebench

Metrics

• average accuracy (primary) — range: [0, 1]
  • Accuracy of correctly identifying the preferred answer in a pairwise comparison. Evaluated by running the model on both original and reversed answer orders independently, then averaging the accuracy across both runs. Parsing failures are counted as incorrect.

Input / output format

Input: Multimodal prompt containing an image, a question, and two candidate answers (Answer A and Answer B) in a target language. Models are evaluated twice per instance: once with the original order and once with the answers reversed.

Output: JSON-formatted string containing the predicted winner (e.g., 'A' or 'B') and a textual reasoning explanation.

Scoring recipe

correct = 0
total = 0
for instance in dataset:
    pred_orig = parse_json(model.generate(instance, order='original'))
    pred_rev = parse_json(model.generate(instance, order='reversed'))
    correct += (pred_orig == instance.gold) + (pred_rev == instance.gold)
    total += 2
return correct / total

Common pitfalls

• If the model's output cannot be parsed into the expected JSON format, the instance is automatically scored as incorrect.
• Positional bias can artificially inflate accuracy if only the original answer order is evaluated; reversing the order is required for robust measurement.
• LaBSE scores for reasoning quality are only reported on the M-OpenCQA subset, not the full benchmark.

Evidence (verbatim from paper)

To ensure robustness, we generate the judgment by providing the candidate answers in both the original and the reversed order independently, and then report the average accuracy. In addition, we evaluate biases like positional bias and length bias. As the LVLMs generate the winner alongside the reasoning, we parse the LVLM-judge predicted winner from their corresponding JSON-formatted responses using a parsing script Laskar et al. (2024). If the parsing script cannot properly parse the judgment from the response, we consider the LVLM-generated answer wrong.

Citation

@misc{laskar2026lost,
  title={Lost in Translation: Do LVLM Judges Generalize Across Languages?},
  author={Md Tahmid Rahman Laskar et al.},
  year={2026},
  note={arXiv:2604.19405}
}

• arXiv: 2604.19405