Vista Multimodal Eval

vista-multimodal-eval

VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization — Li et al. (2025) (arXiv:2505.10917, 2025)

What this evaluates

Evaluates cross-modal vision-text alignment in Multimodal Large Language Models (MLLMs) across high-level semantic VQA, general multimodal understanding, and fine-grained visual perception/retrieval tasks.

Datasets

• VQAv2 — total ?; splits: test (-1)
• OK-VQA — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• RealWorldQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• SEED — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• RefCOCO — total ?; splits: test (-1)
• RefCOCO+ — total ?; splits: test (-1)
• RefCOCOg — total ?; splits: test (-1)

Metrics

• performance (primary) — range: percent
  • Percentage of correct predictions or benchmark-specific score. Aggregated as average improvement over baseline. Evaluated via lmms-eval framework.

Input / output format

Input: Paired image and text prompt/question.

Output: Generated text response or selected multiple-choice option.

Scoring recipe

def compute_performance(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if normalize_answer(p) == normalize_answer(g))
    return correct / len(gold) * 100
# For MME/RefCOCO, use lmms-eval built-in scorers which follow standard benchmark protocols.

Common pitfalls

• RefCOCO performance is sensitive to query length; shorter queries may show dips while longer ones improve.
• Benchmarks like AI2D and MMBench-cn rely heavily on text understanding, so vision-text alignment methods may cause slight performance drops compared to text-only baselines.
• MME reports separate Perception and Cognition scores; aggregating them incorrectly can misrepresent model capabilities.

Evidence (verbatim from paper)

VISTA consistently improves performance on high-level semantic VQA tasks, validating its effectiveness in enhancing vision-text alignment for complex reasoning. As shown in Table[1], integrating our proposed VISTA alignment objective into both TinyLLaVA-3B and LLaVA-v1.5-7B models yields significant performance gains over their original counterparts on high-level VQA datasets.

Citation

@misc{li2025vista,
  title={VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2505.10917}
}

• arXiv: 2505.10917