Basque Multimodal Eval

basque-multimodal-eval

Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque — Arana et al. (2025) (arXiv:2511.09396, 2025)

What this evaluates

Evaluates multimodal large language models on close-ended visual question answering and open-ended generation tasks in Basque and English. It probes visual reasoning, language proficiency, and the impact of training data composition and backbone LLM choice on low-resource language performance.

Datasets

• VQAv2 — total ?; splits: test (-1)
• A-OKVQA — total ?; splits: test (-1)
• PixMoCapQA — total ?; splits: test (-1)
• BertaQA — total ?; splits: test (-1)
• Wildvision — total 199; splits: test (199)

Metrics

• Accuracy (primary) — range: [0, 1]
  • Percentage of correct answers across multiple-choice or close-ended visual question answering benchmarks. Calculated as correct predictions divided by total predictions.
• Winning % — range: percent
  • Percentage of pairwise comparisons where one model's open-ended response is preferred over another's. Calculated as (number of wins / total comparisons) * 100.

Input / output format

Input: Image and question prompt (close-ended); Image, question, and two model responses (open-ended judge evaluation).

Output: Multiple-choice answer or short text (close-ended); Open-ended text response (open-ended).

Scoring recipe

# Close-ended (Accuracy)
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(predictions)

# Open-ended (Winning %)
wins = sum(1 for vote in pairwise_votes if vote == preferred_model)
winning_pct = (wins / len(pairwise_votes)) * 100

Common pitfalls

• LLM-as-a-judge (GPT-5) produces significantly fewer ties than human annotators, leading to biased win rates if used in isolation.
• Training with 0% English data causes catastrophic forgetting on English benchmarks, so a small English mix is required for balanced bilingual performance.
• Close-ended benchmarks may not fully capture language proficiency or open-ended generation quality, necessitating human pairwise evaluation.

Evidence (verbatim from paper)

Latxa and Llama backbone LLMs, obtain a similar average accuracy of 0.62 and 0.61 points across the three benchmarks in both languages, respectively. For each question, the answers generated by both configurations have been compared pairwise to determine the preferred response or whether the result was a tie.

Citation

@misc{arana2025multimodal,
  title={Multimodal Large Language Models for Low-Resource Languages: A Case Study for Basque},
  author={Arana et al. (2025)},
  year={2025},
  note={arXiv:2511.09396}
}

• arXiv: 2511.09396