Captionqa Eval

captionqa-eval

CaptionQA: Is Your Caption as Useful as the Image Itself? — Yang et al. (2025) (arXiv:2511.21025, 2025)

What this evaluates

This benchmark evaluates whether model-generated image captions retain sufficient visual information to answer domain-specific multiple-choice questions without access to the original image. It measures caption utility for downstream reasoning by testing if a text-only QA model can reliably select correct answers or explicitly acknowledge missing information when prompted only with the caption.

Datasets

• CaptionQA — total 33027; splits: full (33027); repo https://github.com/bronyayang/CaptionQA

Metrics

• Caption Utility Score (avg s) (primary) — range: [0, 1]
  • Per-question score s is 1 if the QA LLM selects the correct option, 0 if it selects an incorrect option, and 1/K + 0.05 if it selects 'Cannot answer from the caption.' (where K is the number of semantic options excluding the 'Cannot' choice). The final metric is the average of s across all questions.
• Accuracy (Acc) — range: [0, 1]
  • Fraction of questions where the QA LLM selects the ground-truth option, reflecting the correct/incorrect ratio.
• Cannot ratio (Cannot) — range: [0, 1]
  • Fraction of questions where the QA LLM selects 'Cannot answer', reflecting how often the caption fails to provide sufficient information or the QA LLM judges it so.

Input / output format

Input: A domain-specific multiple-choice question (Q) and a generated image caption (C) provided as text-only input to a QA LLM.

Output: A single selected option from the multiple-choice list, which includes the ground-truth answer, incorrect distractors, and an explicit 'Cannot answer from the caption.' option.

Scoring recipe

def compute_caption_utility_score(predictions, options_per_question, correct_answers):
    total_score = 0.0
    for pred, opts, gold in zip(predictions, options_per_question, correct_answers):
        if pred == gold:
            s = 1.0
        elif pred == "Cannot answer from the caption.":
            K = len(opts) - 1  # exclude Cannot option
            s = (1.0 / K) + 0.05
        else:
            s = 0.0
        total_score += s
    return total_score / len(predictions)

Common pitfalls

• Failing to append the 'Cannot answer from the caption.' option to every non-yes/no question, which breaks the scoring formula and inflates accuracy.
• Not shuffling the option order before prompting the QA LLM, which introduces positional bias and artificially inflates scores for top-listed options.
• Using a QA LLM that hallucinates answers when given an empty or missing caption, violating the faithfulness requirement and corrupting the Cannot ratio baseline.

Evidence (verbatim from paper)

To capture the third case explicitly, we append an additional option “Cannot answer from the caption.” to every non-yes/no question. Then, we have two metrics: Accuracy (Acc). Fraction of questions where the QA LLM selects the ground-truth option, reflecting correct/incorrect ratio. Cannot ratio (Cannot). Fraction of questions where the QA LLM selects “Cannot answer”, reflecting how often the caption fails to provide sufficient information (or the QA LLM judges it so). To summarize both correctness and informative coverage in a single number, we define a per-question score as: s = {1, if selection is correct; 0, if selection is incorrect; 1/K + 0.05, if “Cannot answer from the caption.”}, where K is the number of semantic options (excluding the “Cannot” choice). The final score reported in our tables is the average of s over all questions.

Citation

@misc{yang2025captionqa,
  title={CaptionQA: Is Your Caption as Useful as the Image Itself?},
  author={Yang et al. (2025)},
  year={2025},
  note={arXiv:2511.21025}
}

• arXiv: 2511.21025