M3it Eval

m3it-eval

M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning — Lei Li et al. (2023) (arXiv:2306.04387, 2023)

What this evaluates

Evaluates a vision-language model's ability to follow multi-modal instructions, answer knowledge-based visual questions, and generalize to unseen languages and video tasks. It probes cross-modal alignment, cross-lingual transfer, and the model's conversational response quality.

Datasets

• M^3IT — total 2400000; splits: train (-1), test (-1)
• OK-VQA — total ?; splits: test (-1)
• A-OKVQA — total ?; splits: test (-1)
• ViQuAE — total ?; splits: test (-1)
• Flickr-8k-CN — total ?; splits: test (-1)
• FM-IQA — total ?; splits: test (-1)
• Chinese-FoodNet — total ?; splits: test (-1)
• MSRVTT — total ?; splits: test (-1)
• iVQA — total ?; splits: test (-1)
• ActivityNet-QA — total ?; splits: test (-1)
• MSRVTT-QA — total ?; splits: test (-1)
• MSVD-QA — total ?; splits: test (-1)

Metrics

• ROUGE-L (primary) — range: [0, 1]
  • Computes the longest common subsequence (LCS) based F1 score between the predicted text and the ground-truth answer, focusing on recall of the reference.
• GPT-4 rating — range: [1, 10]
  • GPT-4 rates model responses on a scale of 1-10 based on accuracy, relevance, and naturalness against a reference answer. Winning rate is calculated as the percentage of samples where the model outperforms a baseline.

Input / output format

Input: Image (or uniformly sampled 8 frames for video) paired with a text instruction or question.

Output: Text response or answer generated by the model.

Scoring recipe

# ROUGE-L calculation
rouge_l = compute_rouge_l(predictions=pred_text, references=gold_text)

# GPT-4 proxy evaluation
prompt = f"Question: {q}\nReference: {ref}\nModel A: {pred}\nModel B: {baseline}\nRate 1-10 on accuracy, relevance, naturalness."
rating = gpt4_api.query(prompt).score
winning_rate = sum(1 for r in ratings if r['model'] > r['baseline']) / len(ratings) * 100

Common pitfalls

• ROUGE-L focuses on n-gram overlap and may penalize semantically correct but lexically different responses, failing to capture full conversational quality.
• GPT-4 evaluation is based on only 300 randomly sampled examples, which may not generalize to the full test set distribution.
• Video tasks require uniform sampling of 8 frames, which is a specific preprocessing step that affects temporal context and must be replicated exactly.

Evidence (verbatim from paper)

We adopt ROUGE-L [26] as an automatic metric to assess the consistency between predictions and ground-truth answers, focusing on evaluating the model's conversational abilities. As the automatic metric may not fully capture the nuances of conversational quality, we further introduce GPT-4 as a proxy of human evaluators (§ 4.2).

Citation

@misc{li2023m3it,
  title={M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning},
  author={Lei Li et al. (2023)},
  year={2023},
  note={arXiv:2306.04387}
}

• arXiv: 2306.04387