Maxm Eval

maxm-eval

MaXM: Towards Multilingual Visual Question Answering — Changpinyo et al. (2022) (arXiv:2209.05401, 2022)

What this evaluates

This benchmark evaluates multilingual visual question answering (mVQA) by testing models on images paired with questions in seven languages. It probes a model's ability to perform cross-lingual visual reasoning and generate accurate text answers without relying on costly human annotation.

Datasets

• MaXM — total ?; splits: test (-1); repo https://github.com/google-research-datasets/maxm

Metrics

• Exact Match Accuracy (primary) — range: [0, 1]
  • The prediction is deemed correct if it exactly matches any of the provided ground-truth answers.
• CIDEr — range: [0, 1]
  • An n-gram based similarity metric that weights n-grams by their IDF scores across references. Each ground-truth answer is treated as a separate reference.
• ROUGE-L — range: [0, 1]
  • Measures the longest common subsequence between the prediction and reference answers. Each ground-truth answer is treated as a separate reference.

Input / output format

Input: An image and a natural language question in one of seven target languages (en, fr, hi, iw, ro, th, zh).

Output: A single text string representing the predicted answer.

Scoring recipe

def score_exact_match(predictions, gold_answers):
    correct = 0
    for pred, golds in zip(predictions, gold_answers):
        if pred in golds:
            correct += 1
    return correct / len(predictions)

def score_soft_metrics(predictions, gold_answers, metric_fn):
    scores = []
    for pred, golds in zip(predictions, gold_answers):
        # Treat each GT answer as a separate reference
        score = max(metric_fn(pred, [g]) for g in golds)
        scores.append(score)
    return sum(scores) / len(scores)

Common pitfalls

• MaXM is strictly a test-only benchmark; using it for training violates the protocol.
• Soft metrics like CIDEr and ROUGE-L can mask performance gaps in certain languages (e.g., Thai, Chinese) compared to strict Exact Match Accuracy.
• Applying English-only VQA models directly to non-English questions (Translate-Test) often fails due to tokenizer/vocabulary mismatches, requiring careful adaptation or Translate-Train strategies.

Evidence (verbatim from paper)

We use Exact Match Accuracy as the main evaluation measure for MaXM, following previous work on VQA... We deem the answer as correct if it matches any of the ground-truth answers. To assess the degree of strictness of this measure, we also consider soft text similarity metrics CIDEr and ROUGE-L in our experiments, where we treat each of the ground-truth answers equally as one of the references (as if each of them was answered by an annotator).

Citation

@misc{changpinyo2022maxm,
  title={MaXM: Towards Multilingual Visual Question Answering},
  author={Changpinyo et al. (2022)},
  year={2022},
  note={arXiv:2209.05401}
}

• arXiv: 2209.05401