Caprl Eval

caprl-eval

CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning — Long Xing et al. (2025) (arXiv:2509.22647, 2025)

What this evaluates

Evaluates the quality of dense image captions by measuring how well they enable downstream multimodal models to answer visual questions accurately. It probes fine-grained visual perception, structured description capability, and the utility of captions for non-visual reasoning.

Datasets

• InfoVQA — total ?; splits: test (-1)
• DocVQA — total ?; splits: test (-1)
• ChartQA — total ?; splits: test (-1)
• Real World QA — total ?; splits: test (-1)
• Math Vista — total ?; splits: test (-1)
• SEED2 Plus — total ?; splits: test (-1)
• MME — total ?; splits: test (-1)
• MMB — total ?; splits: test (-1)
• MMStar — total ?; splits: test (-1)
• MMVet — total ?; splits: test (-1)
• AI2D — total ?; splits: test (-1)
• GQA — total ?; splits: test (-1)
• MMMU — total ?; splits: test (-1)
• WeMath — total ?; splits: test (-1)

Metrics

• accuracy (primary) — range: percent
  • Standard multiple-choice or open-ended accuracy: the percentage of correctly answered questions or matched captions. For Prism evaluation, it is the MCQ accuracy of a non-visual LLM answering based solely on the generated caption.

Input / output format

Input: Image (for pretraining eval) or Image + Question (for Prism/VQA eval).

Output: Dense caption text (for pretraining eval) or multiple-choice answer letter/text (for Prism/VQA eval).

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = 0
    for pred, gold_item in zip(predictions, gold):
        if normalize(pred) == normalize(gold_item):
            correct += 1
    return (correct / len(predictions)) * 100

Common pitfalls

• Using subjective LVLM-as-a-judge rewards causes reward hacking, where models generate overly short or verbose captions that satisfy the judge but lack factual utility.
• Option bias in LLMs can distort accuracy scores if questions are not shuffled or evaluated over multiple sampling rounds (N≥4 recommended).
• Averaging across diverse benchmarks (charts, documents, natural images) can mask domain-specific failures; always check per-domain scores.

Evidence (verbatim from paper)

In our setup, we fix Stage 2 with a fine-tuned Qwen2.5-3B-Instruct as the answering LLM, ensuring that benchmark performance directly reflects the quality of captions produced by the captioner. ... Due to inherent option biases in the LLM, the measured accuracy fails to serve as a reliable proxy for reward, thereby misdirecting the optimization of the policy model.

Citation

@misc{xing2025caprl,
  title={CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning},
  author={Long Xing et al. (2025)},
  year={2025},
  note={arXiv:2509.22647}
}

• arXiv: 2509.22647