Multimodal Understanding Eval

multimodal_understanding_eval

InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation — Yang et al. (2025) (arXiv:2507.17520, 2025)

What this evaluates

Evaluates a model's ability to understand and reason over diverse visual inputs, including general VQA, document/chart understanding, OCR, and hallucination robustness.

Datasets

• MMMU(Val) — total ?; splits: val (-1)
• MMStar — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• HallB(Avg) — total ?; splits: test (-1)
• MMB(Dev En V1.1) — total ?; splits: dev (-1)
• TextVQA — total ?; splits: test (-1)
• DoCVQA — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• RWQA — total ?; splits: test (-1)

Metrics

• VLMEvalKit score (primary) — range: percent
  • Percentage of correctly answered questions or matched options, computed automatically via VLMEvalKit.

Input / output format

Input: Image(s) and text prompt/question.

Output: Text response (answer or reasoning).

Scoring recipe

predictions, gold = load_data(benchmark)
score = sum(1 for p, g in zip(predictions, gold) if p == g) / len(gold) * 100
return score

Common pitfalls

• Benchmarks are evaluated using VLMEvalKit, so manual scoring may differ from reported numbers.
• Some benchmarks like MME and HallB report multiple sub-scores; the paper uses averaged or specific sub-scores (e.g., HallB Avg).

Evidence (verbatim from paper)

We use the automatic evaluation from VLMEvalKit [65] including MMMU(Val) [66], MMStar [67], MME [68], OCRBench [69], HallB(Avg) [70], MMB(Dev En V1.1) [71], TextVQA [72], DoCVQA [73], InfoVQA [74], AI2D [75], ChartQA [76] and RWQA [77].

Citation

@misc{yang2025instructvla,
  title={InstructVLA: Vision-Language-Action Instruction Tuning from Understanding to Manipulation},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2507.17520}
}

• arXiv: 2507.17520