Llava Onevision 1.5 Eval

llava-onevision-1.5-eval

LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training — Xiang An et al. (arXiv:2509.23661, 2025)

What this evaluates

This evaluation probes the multimodal reasoning, visual question answering, OCR, and chart understanding capabilities of large multimodal models. It tests the model's ability to process high-resolution images, extract fine-grained text, and perform complex reasoning across diverse visual domains.

Datasets

• MMStar — total ?; splits: test (-1)
• MMEBench — total ?; splits: test (-1)
• MME-RealWorld — total ?; splits: test (-1)
• SeedBench — total ?; splits: test (-1)
• CV-Bench — total ?; splits: test (-1)
• RealWorldQA — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)
• MathVision — total ?; splits: test (-1)
• MMMU — total ?; splits: val (-1)
• MMMU-Pro — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• CharXiv — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• PixmoCount — total ?; splits: test (-1)
• CountBench — total ?; splits: test (-1)
• VL-RewardBench — total ?; splits: test (-1)
• V* — total ?; splits: test (-1)

Metrics

• accuracy / benchmark-specific score (primary) — range: percent
  • Standard exact-match or normalized accuracy computed per benchmark, aggregated using the LMMs-Eval framework. Scores are reported as percentages.

Input / output format

Input: Image and text prompt (question or instruction) provided to the model.

Output: Text response (answer string).

Scoring recipe

def compute_metric(predictions, golds):
    correct = 0
    for pred, gold in zip(predictions, golds):
        pred_clean = normalize_whitespace(pred)
        gold_clean = normalize_whitespace(gold)
        if pred_clean == gold_clean or matches_regex(pred_clean, gold_clean):
            correct += 1
    return (correct / len(golds)) * 100

Common pitfalls

• Using non-default prompts or temperature settings can drastically alter scores on benchmarks like MME or MathVista.
• OCR benchmarks are highly sensitive to image resolution and tiling strategies; inconsistent preprocessing leads to unfair comparisons.

Evidence (verbatim from paper)

We use LMMs-Eval Zhang et al. (2024) with the default prompt to evaluate the performance of LLaVA-OneVision-1.5 across multiple benchmarks in four categories of downstream tasks:(1) General Visual Question Answering (VQA): MMStar (Chen et al., 2024a), MMEBench series (Fu et al., 2023), MME-RealWorld series (Zhang et al., 2025), SeedBench (Li et al., 2024b), Seed-Bench-2-Plus (Li et al., 2024a), CV-Bench (Tong et al., 2024), and RealWorldQA (Corp., 2024). (2) Multimodal Reasoning: MathVista (Lu et al., 2024), WeMath (Qiao et al., 2025), MathVision (Wang et al., 2024a), MMMU (Yue et al., 2024), and MMMU-Pro series (Yue et al., 2025).

Citation

@misc{an2025llavaonevision15,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={Xiang An et al.},
  year={2025},
  note={arXiv:2509.23661}
}

• arXiv: 2509.23661