Mova Eval

mova-eval

MoVA: Adapting Mixture of Vision Experts to Multimodal Context — Zong et al. (2024) (arXiv:2404.13046, 2024)

What this evaluates

Evaluates multimodal large language models' capabilities across general visual question answering, text-oriented VQA (charts, documents, diagrams), visual grounding (referring expression comprehension), and specialized medical VQA. It also assesses general multimodal reasoning and hallucination resistance.

Datasets

• MME — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MMBench-CN — total ?; splits: test (-1)
• QBench — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• MathVerse — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• VQAv2 — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• SQA-I — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• RefCOCO — total ?; splits: val (-1), test-A (-1), test-B (-1)
• RefCOCO+ — total ?; splits: val (-1), test-A (-1), test-B (-1)
• RefCOCOg — total ?; splits: val (-1), test (-1)
• VQA-RAD — total ?; splits: Open (-1), Close (-1)
• SLAKE — total ?; splits: Open (-1), Close (-1)

Metrics

• Accuracy (primary) — range: percent
  • Percentage of correctly answered questions or correctly localized bounding boxes. Calculated as (number of correct predictions / total number of samples) * 100.

Input / output format

Input: Paired image and text instruction/question.

Output: Text answer for VQA tasks; bounding box coordinates for Referring Expression Comprehension (REC) tasks.

Scoring recipe

def calculate_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return (correct / len(gold)) * 100

Common pitfalls

• RefCOCO benchmarks use multiple distinct splits (val, test-A, test-B) that must be reported separately.
• Medical VQA datasets (VQA-RAD, SLAKE) have both 'Open' and 'Close' settings that yield different scores.
• Some baselines are evaluated zero-shot (marked with *) while others are fine-tuned, requiring careful comparison.

Evidence (verbatim from paper)

We develop 105 human-verified testing samples that should be answered using novel experts for the expert routing task. These novel experts encompass 7 vision models [93, 71, 94, 91, 95, 54, 96] on various computer vision tasks and each expert corresponds to 15 evaluation samples. We manually check the correctness of the expert routing result. As presented in Table 10, a lightweight network, such as a MLP classifier fails to generalize to this open-world setting.

Citation

@misc{zong2024mova,
  title={MoVA: Adapting Mixture of Vision Experts to Multimodal Context},
  author={Zong et al. (2024)},
  year={2024},
  note={arXiv:2404.13046}
}

• arXiv: 2404.13046