Aquila Vl Eval

aquila-vl-eval

Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data — Gu et al. (2024) (arXiv:2410.18558, 2024)

What this evaluates

Evaluates a 2B-parameter vision-language model's visual understanding, knowledge reasoning, and text reading capabilities across a comprehensive suite of standard multimodal benchmarks. It measures how well the model handles general VQA, mathematical reasoning, and document comprehension tasks.

Datasets

• MMBench — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMMU — total ?; splits: val (-1)
• MathVista — total ?; splits: testmini (-1)
• HallusionBench — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• MMVet — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard exact-match or multiple-choice accuracy computed per benchmark. Scores are reported as percentages averaged across the listed benchmarks.

Input / output format

Input: Paired image and text instruction/question.

Output: Text response (multiple-choice selection or open-ended answer).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p.strip().lower() == g.strip().lower())
    return (correct / len(gold)) * 100

Common pitfalls

• Benchmarks use heterogeneous evaluation protocols (e.g., multiple-choice vs. open-ended), making direct score comparison difficult.
• The reported 'Average' is a simple arithmetic mean across disparate benchmarks, which can mask performance on specific capabilities.
• Some benchmarks (e.g., MMVet, MME) have strict and lenient evaluation modes; the paper does not specify which mode was used.

Evidence (verbatim from paper)

We assessed the visual capabilities of Aquila-VL-2B using a range of visual benchmarks provided by the VLMEvalKit*[[21]]*. Experimental results are shown in Table [4]. Aquila-VL-2B demonstrates highly competitive performance at the same scale, achieving new state-of-the-art results.

Citation

@misc{gu2024infinitymm,
  title={Infinity-MM: Scaling Multimodal Performance with Large-Scale and High-Quality Instruction Data},
  author={Gu et al. (2024)},
  year={2024},
  note={arXiv:2410.18558}
}

• arXiv: 2410.18558