Mtvqa Eval

mtvqa-eval

MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering — Tang et al. (2024) (arXiv:2405.11985, 2024)

What this evaluates

This benchmark evaluates the multilingual visual-textual alignment and comprehension capabilities of multimodal large language models (MLLMs). It specifically probes whether models can accurately perceive, extract, and reason about text embedded within images across nine different languages without relying on translation.

Datasets

• MTVQA — total ?; splits: train (-1), test (-1); repo https://github.com/bytedance/MTVQA

Metrics

• Accuracy (primary) — range: percent
  • Measures the percentage of questions for which the predicted answer matches any of the target answers for the question.

Input / output format

Input: An image containing embedded text and a question in one of nine languages (AR, DE, FR, IT, JA, KO, RU, TH, VI). Models are prompted with: 'Answer the question using a word or phrase in the language of the question. + '.

Output: A concise word or phrase in the language of the question.

Scoring recipe

def calculate_accuracy(predictions, gold_answers):
    correct = 0
    for pred, golds in zip(predictions, gold_answers):
        if pred.strip().lower() in [g.strip().lower() for g in golds]:
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Models show significant performance drops on non-Latin scripts (AR, JA, KO, TH) due to weaker visual text perception compared to Latin-based languages.
• Using external OCR pipelines before feeding text to LLMs/MLLMs degrades performance because OCR strips crucial positional and visual context.
• Relying on the ANLS metric can overestimate model performance by 5-10% compared to exact Accuracy, masking failures in precise text extraction.

Evidence (verbatim from paper)

To accurately assess whether the visual text that occurs in the answer is correct, we adopt Accuracy as the metric. The Accuracy metric measures the percentage of questions for which the predicted answer matches any of the target answers for the question.

Citation

@misc{tang2024mtvqa,
  title={MTVQA: Benchmarking Multilingual Text-Centric Visual Question Answering},
  author={Tang et al. (2024)},
  year={2024},
  note={arXiv:2405.11985}
}

• arXiv: 2405.11985