Compcap Eval

compcap-eval

CompCap: Improving Multimodal Large Language Models with Composite Captions — Chen et al. (2024) (arXiv:2412.05243, 2024)

What this evaluates

Evaluates Multimodal Large Language Models' ability to comprehend composite images (charts, collages, tables, code) and natural images, covering text recognition, visual reasoning, and conversational capabilities.

Datasets

• SEEDBench* — total ?; splits: test (-1)
• TextVQA — total ?; splits: test (-1)
• MMBench — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• LLaVABench — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• InfoVQA — total ?; splits: test (-1)
• WebSRC — total ?; splits: test (-1)
• MathVista — total ?; splits: test (-1)
• OCRBench — total ?; splits: test (-1)

Metrics

• Average score (primary) — range: percent
  • Mean of per-benchmark accuracy percentages across all 11 evaluation sets.

Input / output format

Input: Multimodal input consisting of an image and a text prompt/question.

Output: Text response (answer, caption, or classification label).

Scoring recipe

def compute_score(predictions, golds):
    correct = sum(1 for p, g in zip(predictions, golds) if p.strip().lower() == g.strip().lower())
    return (correct / len(golds)) * 100

# Per benchmark accuracy
benchmark_scores = [compute_score(preds, golds) for preds, golds in zip(all_predictions, all_golds)]
# Headline metric
headline_metric = sum(benchmark_scores) / len(benchmark_scores)

Common pitfalls

• Benchmarks contain mixed natural images (NIs) and composite images (CIs); results should be stratified by image type for fair analysis.
• Evaluation relies on automated parsing tools (VLMEvalKit, LMMs-EVAL) which may differ from manual grading or have parsing edge cases.
• Compared models are trained on vastly different data scales (10x-100x differences), so raw score comparisons can be misleading without accounting for training data volume.

Evidence (verbatim from paper)

We report the average scores over NI-dominated benchmarks (SEEDBench, TextVQA, MMBench, MME, LLaVABench), CI-dominated benchmarks (MathVista, OCRBench, ChartQA, DocVQA, InfoVQA, WebSRC), and all benchmarks.

Citation

@misc{chen2024compcap,
  title={CompCap: Improving Multimodal Large Language Models with Composite Captions},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2412.05243}
}

• arXiv: 2412.05243