Blip3 Multimodal Benchmarks Eval

blip3-multimodal-benchmarks-eval

xGen-MM (BLIP-3): A Family of Open Large Multimodal Models — Le Xue et al. (2024) (arXiv:2408.08872, 2024)

What this evaluates

Evaluates large multimodal models on single- and multi-image understanding, covering general VQA, domain knowledge, OCR, hallucination, and interleaved image-text reasoning.

Datasets

• SEED-IMG — total ?; splits: test (-1)
• MMB(dev) — total ?; splits: dev (-1)
• MMStar — total ?; splits: test (-1)
• MME(norm) — total ?; splits: test (-1)
• RWQA — total ?; splits: test (-1)
• MMVet — total ?; splits: test (-1)
• MMMU(val) — total ?; splits: val (-1)
• MathVista — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)
• POPE — total ?; splits: test (-1)
• HalBench — total ?; splits: test (-1)
• BLINK — total ?; splits: test (-1)
• QBench — total ?; splits: test (-1)
• MuirBench — total ?; splits: test (-1)
• Mantis-Eval — total ?; splits: test (-1)

Metrics

• benchmark score (primary) — range: percent
  • Standard accuracy or benchmark-specific scoring metric (e.g., exact match, LLM-as-judge) applied per dataset. Values are reported as percentages.
• average score (primary) — range: percent
  • Arithmetic mean of the individual benchmark scores across all listed single- and multi-image benchmarks.

Input / output format

Input: Single image or interleaved multi-image(s) paired with text prompts/questions.

Output: Text response (answer to the prompt/question).

Scoring recipe

scores = []
for dataset in BENCHMARKS:
    pred = model.generate(image, prompt)
    score = dataset.evaluate(pred, gold)  # Standard per-dataset metric
    scores.append(score)
average_score = sum(scores) / len(scores)
return average_score

Common pitfalls

• Mixing up single-image (SI) and multi-image (MI) model variants when reporting scores, as the paper explicitly compares them.
• Using third-party leaderboard scores instead of official evaluation code, which the paper notes is necessary for fair comparison.
• Assuming a uniform scoring formula across all benchmarks, as each uses its own standard metric (e.g., MMVet uses LLM-as-judge, while others use exact match).

Evidence (verbatim from paper)

We evaluate our models (4B and 14B) on a comprehensive suite of multimodal benchmarks, assessing the model’s ability from multiple perspectives. Our evaluation covers general VQA benchmarks, domain knowledge, OCR ability, and hallucination. For models fine-tuned on interleaved multi-image datasets, we also evaluate their performance on common multi-image benchmarks. ... We report individual benchmark scores as well as the overall average score across all benchmarks, following the standard practice.

Citation

@misc{xue2024xgenmm,
  title={xGen-MM (BLIP-3): A Family of Open Large Multimodal Models},
  author={Le Xue et al. (2024)},
  year={2024},
  note={arXiv:2408.08872}
}

• arXiv: 2408.08872