Eagle2 Vlm Eval

eagle2-vlm-eval

Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models — Li et al. (2025) (arXiv:2501.14818, 2025)

What this evaluates

Evaluates vision-language models on document understanding, chart and table reasoning, OCR, diagram comprehension, and general visual question answering. The protocol measures accuracy across a diverse suite of 14 established multimodal benchmarks to assess overall multimodal capability and robustness.

Datasets

• DocVQA — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• MMMU — total ?; splits: val (-1)
• MMB1.1 — total ?; splits: test (-1)
• MathVista — total ?; splits: test-mini (-1)

Metrics

• OpenCompass (primary) — range: percent
  • Arithmetic mean of scores across OCRBench, MMStar, AI2D, MMMU, MMB1.1, MMVet, HallusionBench, and MathVista.
• Benchmark Accuracy — range: percent
  • Per-benchmark accuracy score. Computed using each dataset's official evaluation script (typically exact match for VQA or LLM-as-a-judge for open-ended tasks).

Input / output format

Input: A single image paired with a natural language question or instruction prompt.

Output: A natural language text string containing the model's predicted answer.

Scoring recipe

def evaluate_model(model, dataset_splits):
    all_scores = []
    for split in dataset_splits:
        preds = model.generate(split.images, split.prompts)
        score = official_scorer(split.golds, preds)
        all_scores.append(score)
    opencompass_score = sum(all_scores) / len(all_scores)
    return opencompass_score, all_scores

Common pitfalls

• Benchmarks use different evaluation splits (Test, Val, Test-Mini); the evaluation script must strictly match the split reported in the paper.
• OpenCompass is a custom composite metric averaging 8 specific benchmarks, not a standard public leaderboard score.
• Some models have missing scores on certain benchmarks (e.g., InfoVQA, RWQA), which must be handled as NaN rather than zero when computing averages.

Evidence (verbatim from paper)

*: We obtain the OpenCompass score by averaging across Eagle benchmarks (OCRBench, MMStar, AI2D, MMMU, MMB1.1, MMVet, HallusionBench, and MathVista).

Citation

@misc{li2025eagle2,
  title={Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2501.14818}
}

• arXiv: 2501.14818