qhjqhj00 mousi-vlm-eval
MouSi: Poly-Visual-Expert Vision-Language Models — Xiaoran Fan et al. (2024) (arXiv:2401.17221, 2024)
What this evaluates
Evaluates multimodal understanding and reasoning across visual question answering, OCR, region-level VQA, and visual conversation tasks. It probes how effectively poly-visual expert ensembles fuse information from multiple encoders compared to single-expert baselines.
Datasets
- LLaVA-1.5 Benchmark Suite — total ?; splits: test (-1)
Metrics
accuracy(primary) — range: percent- Percentage of correctly answered questions or selected options per benchmark, averaged across the 9 evaluation datasets.
Input / output format
Input: Image and a natural language question or prompt.
Output: Natural language text response or multiple-choice option.
Scoring recipe
def compute_accuracy(predictions, gold):
correct = 0
for pred, gold in zip(predictions, gold):
if normalize_answer(pred) == normalize_answer(gold):
correct += 1
return (correct / len(gold)) * 100
Common pitfalls
- Excluded LLaVA-Bench, VisWiz, and MME due to instability or website crashes, limiting full reproducibility.
- Performance is highly sensitive to fusion architecture (MLP vs Q-Former) and expert ordering, which vary across ablations.
- Triple-expert models hit performance ceilings likely due to limited pre-training data (665K samples), suggesting data scaling is critical.
Evidence (verbatim from paper)
We selected 9 of the 12 evaluation benchmarks for LLaVA-1.5 (excluding LLaVA-Bench that rely on unstable GPT4 responses, as well as VisWiz and MME for the website crashed), including VQA${}^{ ext{v2}}$; GQA; SQA${}^{ ext{I}}$ : ScienceQA-IMG; VQA${}^{ ext{T}}$: TextVQA; POPE; MMB & MMB${}^{ ext{CN}}$: MMBench & MMBench-Chinese dev results; SEED${}^{ ext{I}}$ : SEED-Bench-IMG; MM-Vet.
Citation
@misc{fan2024mousi,
title={MouSi: Poly-Visual-Expert Vision-Language Models},
author={Xiaoran Fan et al. (2024)},
year={2024},
note={arXiv:2401.17221}
}
- arXiv: 2401.17221