Cvqa Eval

cvqa-eval

CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark — Romero et al. (2024) (arXiv:2406.05967, 2024)

What this evaluates

This benchmark evaluates the cultural and linguistic understanding of multimodal vision-language models by testing their ability to answer multiple-choice questions about images in diverse languages and cultural contexts. It probes zero-shot generalization across location-aware and location-agnostic prompts, highlighting performance gaps in low-resource languages.

Datasets

• CVQA — total 10000; splits: test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted options out of the total number of questions. Calculated as (number of correct predictions) / (total questions).

Input / output format

Input: An image paired with a text prompt containing a question and four multiple-choice options (A, B, C, or D). Prompts are either location-aware (includes country) or location-agnostic (excludes country), and can be in English or a local language.

Output: A single letter corresponding to the selected option (A, B, C, or D).

Scoring recipe

if model_type == 'generative':
    probs = model.predict_probabilities(image, prompt)
    prediction = argmax(probs['A'], probs['B'], probs['C'], probs['D'])
elif model_type == 'embedding':
    sims = [similarity(model.encode(image), model.encode(f'{question} {option}')) for option in ['A', 'B', 'C', 'D']]
    prediction = argmax(sims)
accuracy = sum(prediction == gold) / len(gold)

Common pitfalls

• Generative and embedding-based models require different scoring logic (probability argmax vs. cross-modal similarity argmax).
• Evaluating location-aware vs. location-agnostic prompts yields different results; mixing them without tracking breaks comparability.
• Multilingual prompts must be matched to the model's supported languages; evaluating low-resource languages with monolingual models will yield near-zero accuracy.

Evidence (verbatim from paper)

We perform a zero-shot evaluation with two types of prompts, as follows: a location-aware prompt, which specifies the country, the question, and the options, (e.g., “Location: {country}. Question: {question} Options: {options} Short Answer:”); and a location-agnostic prompt, which follows the same template but does not specify the country in the prompt (e.g., “Question: {question} Options: {options} Short Answer:”). Additionally, due to the multilingual nature of CVQA, for each prompt, we evaluate using the English-only and local language question-option pairs. For the generative-based models, LLaVA, mBLIP and InstructBLIP, the image and the prompts are used as the input. The models then produce output probabilities and we treat the highest probability for the options (A,B,C,D) as the prediction (following MMLU*[[16]]*). On the other hand, for embedding-based models like CLIP and M-CLIP, we use the embedding-level similarity between the image and the combination of question and each answer candidate texts (Question+Option-1,…,Question+Option-4) to select the one with the highest similarity as the correct answer. We use accuracy to measure the performance, following the existing mul

Citation

@misc{romero2024cvqa,
  title={CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark},
  author={Romero et al. (2024)},
  year={2024},
  note={arXiv:2406.05967}
}

• arXiv: 2406.05967