P2g Visual Reasoning Eval

p2g-visual-reasoning-eval

Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models — Chen et al. (2024) (arXiv:2403.19322, 2024)

What this evaluates

Evaluates the visual reasoning and text-understanding capabilities of multimodal large language models (MLLMs) on high-resolution, text-rich, and general semantic images. It probes whether agent-augmented grounding improves answer accuracy compared to vanilla MLLMs and proprietary models like GPT-4V.

Datasets

• DocVQA — total ?; splits: test (-1)
• ChartVQA — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• SEED — total ?; splits: test (-1)
• MM-VET — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• P2GB — total ?; splits: test (-1)

Metrics

• VQA score (primary) — range: percent
  • Standard dataset-specific evaluation metric (typically exact match or fuzzy match accuracy) used for each visual question answering benchmark. Scores are reported as percentages in the tables.

Input / output format

Input: High-resolution image paired with a natural language question or instruction.

Output: Natural language answer string, optionally accompanied by generated reasoning and bounding box coordinates for grounded objects/texts.

Scoring recipe

def compute_vqa_score(predictions, gold_answers):
    score = 0
    for pred, golds in zip(predictions, gold_answers):
        # Standard VQA evaluation: exact match or fuzzy match (e.g., 3-gram overlap)
        if normalize_answer(pred) in [normalize_answer(g) for g in golds]:
            score += 1
    return (score / len(predictions)) * 100

Common pitfalls

• Benchmark scores are highly sensitive to input resolution and context window limits; high-res images may be truncated or downsampled.
• Text-rich benchmarks require external OCR agents; OCR errors or missed text directly degrade scores.
• Including relative position vectors of grounded boxes in the prompt significantly impacts performance on structured data like charts.

Evidence (verbatim from paper)

Table 5: Experimental results of P2G and baselines on visual reasoning benchmarks. The performance of P2G on visual reasoning benchmarks are presented in Table 6. On text-rich visual reasoning, P2G significantly outperform baselines, including the vanilla LLaVA, by more than doubled (3× on DocVQA, 2.4× on ChartVQA)... On general visual reasoning benchmarks, P2G also enjoys a consistent improvement over LLaVA and InstrtuctBLIP, demonstrating the superiority of P2G.

Citation

@misc{chen2024p2g,
  title={Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2403.19322}
}

• arXiv: 2403.19322