Vlm Benchmarks Eval

vlm-benchmarks-eval

Rethinking Overlooked Aspects in Vision-Language Models — Yuan Liu et al. (arXiv:2405.11850, 2024)

What this evaluates

Evaluates vision-language models on instruction-following and multimodal reasoning tasks across multiple established benchmarks. Probes capabilities in general VQA, mathematical reasoning, scientific understanding, hallucination detection, and multilingual comprehension.

Datasets

• MMBench — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• HallusionBench — total ?; splits: test (-1)
• SEEDBench — total ?; splits: test (-1)
• LLaVABench — total ?; splits: test (-1)
• ScienceQA — total ?; splits: test (-1)

Metrics

• evaluation metric (primary) — range: other
  • Uses the official evaluation metric defined by each benchmark's original paper. If unavailable, the metric from the VLMEvalKit leaderboard is used. Scores are typically accuracy or task-specific scores.

Input / output format

Input: Image and text instruction/prompt provided to the vision-language model.

Output: Text response generated by the model, evaluated against ground truth using each benchmark's official scoring protocol.

Scoring recipe

def compute_metric(predictions, gold, benchmark_name):
    if benchmark_name in official_scorers:
        return official_scorers[benchmark_name].compute(predictions, gold)
    else:
        return vlmevalkit_leaderboard_metric.compute(predictions, gold)

Common pitfalls

• Metrics vary significantly across benchmarks; the paper defers to original papers or VLMEvalKit leaderboards rather than standardizing a single metric.
• Performance on MME is mapped to a 0-100 scale for reporting in ablation studies, which may differ from the raw score used in other contexts.

Evidence (verbatim from paper)

We select the evaluation metric for each method based on its original paper, if available. If not, we use the metric provided by the leaderboard of VLMEvalKit.

Citation

@misc{liu2024rethinking,
  title={Rethinking Overlooked Aspects in Vision-Language Models},
  author={Yuan Liu et al.},
  year={2024},
  note={arXiv:2405.11850}
}

• arXiv: 2405.11850